sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}"
mkoeppe opened this issue · 48 comments
We extend the sage_bootstrap package to add the commands:
sage -package list --output=install-requires PACKAGES...(for #30913)sage -package list --output=requirements PACKAGES...sage -package list --output=pipfile PACKAGES...sage -package list --output=debian PACKAGE...etc. (taking over the functionality of the shell scriptbuild/bin/sage-get-system-packages.)
They generate text output in the format of setup.cfg [options] install_requires=, requirements.txt, and Pipfile; or a list of system packages
bootstrap can use these commands to generate these files for build/pkgs/sagelib/src/ or src. They are deleted by make bootstrap-clean.
References:
- https://pip.pypa.io/en/stable/user_guide/#requirements-files
- https://pipenv-fork.readthedocs.io/en/latest/advanced.html
- https://packaging.python.org/discussions/install-requires-vs-requirements/
- https://pypi.org/project/install-requires/
CC: @seblabbe @dimpase @embray @vbraun @kiwifb @tobihan @tobiasdiez
Component: build
Keywords: sd111
Issue created by migration from https://trac.sagemath.org/ticket/29041
This is what it could look like - just for a small handful of packages for now
New commits:
cfcc9fd | bootstrap: generate src/requirements.txt, src/constraints.txt, src/setup.cfg [install_requires] from build/pkgs |
The apt-get line in the install manual could be generated in the same way.
Author: Matthias Koeppe
I see that only few packages, e.g. pillow, are mentioned in the branch. Are they there as an example?
Yes, it's an initial example. More can be added on this ticket and/or on follow up tickets.
It may be the most efficient to first put this infrastructure (and the one for Debian) in place and then do invidual packages one by one to determine useful version bounds.
how is it going to play out with version constraints in spkg-configure.m4 files?
So far there are no spkg-configure.m4 files for any Python packages, and I would hope there will be no need for them.
Basically sagelib-install-requires.txt will play the same role for Python packages as spkg-configure.m4 does for non-Python packages,
and likewise sagelib-requirements.txt : package-version.txt.
Similar to what I wrote on #26964, I don't think this should be done in the bootstrap script. I don't want this script growing too far beyond its core purpose, which is simply to bootstrap the configure script.
This could be done by configure instead, using say a requirements.txt.in.
Branch pushed to git repo; I updated commit sha1. New commits:
0dc97db | Clean src/doc/en/installation/*.txt in bootstrap-clean, not misc-clean |
bae1a82 | Merge branch 't/29233/bootstrap____make_distclean_broken' into t/29041/at___bootstrap_time__generate_src_requirements_txt__src_constraints_txt__src_setup_cfg__install_requires__from_build_pkgs |
4fb30a6 | bootstrap: Add src/constraints.txt etc. to configure tarball |
Branch pushed to git repo; I updated commit sha1. New commits:
f147962 | src/bootstrap: Make executable |
Description changed:
---
+++
@@ -2,3 +2,5 @@
This is another step toward #21507 - Task ticket: Make sagelib a pip-installable Python source package, listed on PyPI
+Like the files generated in #26964, they are generated at `bootstrap` time so that they are available when a source distribution is packaged. They are deleted by `make bootstrap-clean`.
+Branch pushed to git repo; I updated commit sha1. New commits:
13088e3 | Merge tag '9.1.beta6' into t/29041/at___bootstrap_time__generate_src_requirements_txt__src_constraints_txt__src_setup_cfg__install_requires__from_build_pkgs |
Needs review
Description changed:
---
+++
@@ -4,3 +4,12 @@
Like the files generated in #26964, they are generated at `bootstrap` time so that they are available when a source distribution is packaged. They are deleted by `make bootstrap-clean`.
+The files would be generated either in `build/pkgs/sagelib/src/` or in `src`.
+
+---
+
+References:
+
+https://pipenv-fork.readthedocs.io/en/latest/advanced.html
+
+Another approach would be to maintain an independent pipfile/requirements.txt in src/ and have automatic checks to make sure that the sage packages provide the required packages with a suitable version.
Reason: I feel like sage-lib should be independent of sage-distribution as much as possible.
More practical reason: pipenv works very nicely on Windows, and my hope is to use it without having to use sage make beforehand. So one simply clones the code, starts the virtual env with pipenv and gets an editiable install of sage for easy development. But for this the pipfile needs to be there without running the bootstrap command. (The alternative would be to rewrite the bootstrap cmd in a OS-independent way, say using powershell.)
Replying to @tobiasdiez:
Another approach would be to maintain an independent pipfile/requirements.txt in src/ and have automatic checks to make sure that the sage packages provide the required packages with a suitable version.
Reason: I feel like sage-lib should be independent of sage-distribution as much as possible.
I think the versions given by sage-the-distribution (build/pkgs/*/package-version.txt) should be considered as pinned versions, and information such as requirements.txt or Pipfile/Pipfile.lock should come from there in some way.
The allowed version ranges are currently not encoded anywhere. For non-Python packages have spkg-configure.m4 (#27330), but this is obviously the wrong mechanism for Python packages. setuptools has them in setup.cfg [install_requires].
I think I agree with you that this information does not necessarily have to be generated from (new) files in build/pkgs/ but could be put directly into src/, such as src/setup.cfg.
More practical reason: pipenv works very nicely on Windows, and my hope is to use it without having to use
sage makebeforehand. So one simply clones the code, starts the virtual env with pipenv and gets an editiable install of sage for easy development.
I think some steps are missing here - we don't have a native Windows build of many packages that Sage depends on. But I agree, after modularization (#29705) we should aim for getting a native Windows build of a Sage core at least.
But for this the pipfile needs to be there without running the bootstrap command. (The alternative would be to rewrite the bootstrap cmd in a OS-independent way, say using powershell.)
Note that we have the bootstrap -d mechanism, which downloads a tarball of the files created by bootstrap. This is how most users and developers get these files. We could look into creating a powershell version of bootstrap -d.
Description changed:
---
+++
@@ -10,6 +10,6 @@
References:
-https://pipenv-fork.readthedocs.io/en/latest/advanced.html
+- https://pipenv-fork.readthedocs.io/en/latest/advanced.html
+- https://packaging.python.org/discussions/install-requires-vs-requirements/
-There is also https://pypi.org/project/pipenv-to-requirements/ to generate the requirements file from a pipfile.
Changed author from Matthias Koeppe to none
Description changed:
---
+++
@@ -1,15 +1,18 @@
-https://pip.pypa.io/en/stable/user_guide/#requirements-files
+We extend the `sage_bootstrap` package to add the commands:
+- `sage -package get-system-packages install-requires PACKAGES...` (for #30913)
+- `sage -package get-system-packages requirements PACKAGES...`
+- `sage -package get-system-packages pipfile PACKAGES...`
+(The functionality of the shell script `build/bin/sage-get-system-packages` can be merged into this command as well.)
-This is another step toward #21507 - Task ticket: Make sagelib a pip-installable Python source package, listed on PyPI
+They generate text files in the format of `setup.cfg [options] install_requires=`, `requirements.txt`, and `Pipfile`.
-Like the files generated in #26964, they are generated at `bootstrap` time so that they are available when a source distribution is packaged. They are deleted by `make bootstrap-clean`.
-
-The files would be generated either in `build/pkgs/sagelib/src/` or in `src`.
+`bootstrap` can use these commands to generate these files for `build/pkgs/sagelib/src/` or `src`. They are deleted by `make bootstrap-clean`.
---
References:
+- https://pip.pypa.io/en/stable/user_guide/#requirements-files
- https://pipenv-fork.readthedocs.io/en/latest/advanced.html
- https://packaging.python.org/discussions/install-requires-vs-requirements/
Description changed:
---
+++
@@ -4,7 +4,7 @@
- `sage -package get-system-packages pipfile PACKAGES...`
(The functionality of the shell script `build/bin/sage-get-system-packages` can be merged into this command as well.)
-They generate text files in the format of `setup.cfg [options] install_requires=`, `requirements.txt`, and `Pipfile`.
+They generate text output in the format of `setup.cfg [options] install_requires=`, `requirements.txt`, and `Pipfile`.
`bootstrap` can use these commands to generate these files for `build/pkgs/sagelib/src/` or `src`. They are deleted by `make bootstrap-clean`.
Changed keywords from none to sd111
Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111
Description changed:
---
+++
@@ -1,10 +1,10 @@
We extend the `sage_bootstrap` package to add the commands:
-- `sage -package get-system-packages install-requires PACKAGES...` (for #30913)
-- `sage -package get-system-packages requirements PACKAGES...`
-- `sage -package get-system-packages pipfile PACKAGES...`
-(The functionality of the shell script `build/bin/sage-get-system-packages` can be merged into this command as well.)
+- `sage -package list --output=install-requires PACKAGES...` (for #30913)
+- `sage -package list --output=requirements PACKAGES...`
+- `sage -package list --output=pipfile PACKAGES...`
+- `sage -package list --output=debian PACKAGE...` etc. (taking over the functionality of the shell script `build/bin/sage-get-system-packages`.)
-They generate text output in the format of `setup.cfg [options] install_requires=`, `requirements.txt`, and `Pipfile`.
+They generate text output in the format of `setup.cfg [options] install_requires=`, `requirements.txt`, and `Pipfile`; or a list of system packages
`bootstrap` can use these commands to generate these files for `build/pkgs/sagelib/src/` or `src`. They are deleted by `make bootstrap-clean`.
Rewriting tools used in the bootstrap phase in Python would also be useful for developing on Windows
hmm, what is so problematic about Windows here? Even if one talks about native, non-cygwin, it is still perfectly possible to run bash etc.
Description changed:
---
+++
@@ -15,4 +15,4 @@
- https://pip.pypa.io/en/stable/user_guide/#requirements-files
- https://pipenv-fork.readthedocs.io/en/latest/advanced.html
- https://packaging.python.org/discussions/install-requires-vs-requirements/
-
+- https://pypi.org/project/install-requires/Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
Setting a new milestone for this ticket based on a cursory review.