make sagelib a script package
mkoeppe opened this issue · 78 comments
... so that sagelib is just an ordinary script package whose source tree is included (hence, no checksums.ini file; see #29287).
Related sage-devel discussions:
- https://groups.google.com/d/msg/sage-devel/nTwhCV89FXE/_7GdzGy4BgAJ ("package sources")
- https://groups.google.com/d/msg/sage-devel/nTwhCV89FXE/-obd4rMDBAAJ
Related tickets:
- #29386 Install script packages via sage-spkg
Depends on #29098
Depends on #29697
Depends on #29793
CC: @embray @videlec @orlitzky @dimpase @jhpalmieri @kiwifb
Component: build
Author: Matthias Koeppe
Branch: cc30471
Reviewer: Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/29411
Description changed:
---
+++
@@ -1 +1,10 @@
+... so that sagelib is just an ordinary `script` package whose source tree is included (hence, no `checksums.ini` file; see #29287).
+Related sage-devel discussions:
+- https://groups.google.com/d/msg/sage-devel/nTwhCV89FXE/_7GdzGy4BgAJ ("package sources")
+- https://groups.google.com/d/msg/sage-devel/nTwhCV89FXE/-obd4rMDBAAJ
+
+Related tickets:
+- #29386 Install script packages via sage-spkg
+
+Not quite done yet. Next step is to get rid of src/Makefile.in, moving the contents into the spkg-install and spkg-clean scripts.
The clean target should perhaps be transformed into setup.py clean, reference: pypa/setuptools#1347
Author: Matthias Koeppe
Branch pushed to git repo; I updated commit sha1. New commits:
91e5a3d | src/Makefile.in: Merge into build/make/Makefile.in, build/pkgs/sagelib/spkg-install |
Replying to @mkoeppe:
The
cleantarget should perhaps be transformed intosetup.py clean, reference: pypa/setuptools#1347
That's already noted in #21508, will not happen on the present ticket
TBD: sagelib rebuilds are no longer triggered because it now goes through a non-phony target
Branch pushed to git repo; I updated commit sha1. New commits:
08cbbf2 | build/make/Makefile.in, build/pkgs/sagelib/dependencies: Make sure sagelib spkg-install is always run |
Not quite what I had in mind but this is a step in the right direction. It makes sagelib less special in the build system. I believe the dependency part is OK. The makefile bits deserve a bit scrutiny and field testing.
The old src/Makefile.in and now the new build/pkgs/sagelib/spkg-install has this in it:
Hoping that #20382 will make this unnecessary.
#20382 has been merged, so perhaps this should be clarified, although not necessarily on this ticket.
Branch pushed to git repo; I updated commit sha1. New commits:
5e915b0 | build/pkgs/sagelib/spkg-install: Update references to tickets |
Branch pushed to git repo; I updated commit sha1. New commits:
53f0359 | src/sage/env.py (sage_include_directories): Do not put SAGE_INC in front of the sage source include directories |
16247ad | src/setup.py: Do not put SAGE_LOCAL/lib in front of the library directories |
9a50cba | src/sage/env.py (sage_include_directories): Fixup doctest |
73fb13a | Merge branch 't/29697/src_setup_py__src_sage_env_py__sage_include_directories___do_not_add_another_copy_of_sage_inc__sage_local_lib_to_include_dirs__library_dirs' into t/29411/make_sagelib_a_script_package |
f6e3fa1 | build/pkgs/sagelib/spkg-install: Also poison environment variables SAGE_LOCAL etc. |
Ready for review.
Branch pushed to git repo; I updated commit sha1. New commits:
fcf2cab | build/pkgs/sagelib/spkg-install: Add comment regarding SAGE_SPKG_INST |
Branch pushed to git repo; I updated commit sha1. New commits:
a973fc4 | build/pkgs/sagelib/spkg-install: Also poison SAGE_PKGCONFIG, SAGE_SPKG_SCRIPTS |
please remove gcc from the modified comment in src/setup.py - it's a misleading artefact from the past, where gcc was needed to build Sage.
Branch pushed to git repo; I updated commit sha1. New commits:
c1db7ed | src/setup.py: Update comment (not specific to gcc) |
What is the point of FORCE target - to always (re)build sagelib, right?
But is it needed?
Yes, without it, sagelib would not be rebuilt when source files change.
hmm, but why not honestly list deps in a makefile?
Well, instead of the dependency on FORCE that is listed in build/pkgs/sagelib/dependencies, we could make it a dependency on $(SAGE_SRC) $(SAGE_SRC)/* $(SAGE_SRC)/*/* $(SAGE_SRC)/*/*/* $(SAGE_SRC)/*/*/*/*, but I don't think this is better
In other words, we do not have dependency tracking at all for sagelib. Neither of distutils, setuptools, cython seem to have a mechanism for generating dependencies like gcc -M. This would be certainly be nice to have; in particular if it included dependencies on "system" headers and libraries, so that sagelib could automatically rebuild when system headers/libraries change (as in, most recently, https://groups.google.com/d/msg/sage-release/A53tGGsJNLg/uA01oUugAQAJ). But that's outside of the scope of this ticket.
Reviewer: Dima Pasechnik
OK, makes sense.
Thanks!
I have created #29711 (sagelib setup.py: Dependencies on header files of packages).
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. Last 10 new commits:
e1a9413 | Merge build/make/deps into build/make/Makefile.in |
da9538e | Make sagelib a script package |
9fd3f7e | src/Makefile.in: Merge into build/make/Makefile.in, build/pkgs/sagelib/spkg-install |
9ea77e5 | build/make/Makefile.in, build/pkgs/sagelib/dependencies: Make sure sagelib spkg-install is always run |
c3908e9 | build/pkgs/sagelib/spkg-install: Update references to tickets |
62f2841 | Merge branch 't/29697/src_setup_py__src_sage_env_py__sage_include_directories___do_not_add_another_copy_of_sage_inc__sage_local_lib_to_include_dirs__library_dirs' into t/29411/make_sagelib_a_script_package |
048ce5b | build/pkgs/sagelib/spkg-install: Also poison environment variables SAGE_LOCAL etc. |
4590c4c | build/pkgs/sagelib/spkg-install: Add comment regarding SAGE_SPKG_INST |
ed312c5 | build/pkgs/sagelib/spkg-install: Also poison SAGE_PKGCONFIG, SAGE_SPKG_SCRIPTS |
6ee66dd | src/setup.py: Update comment (not specific to gcc) |
Oh, I was just thinking about this. Thanks for picking up the work. This is something I've been wanting to do for years.
I'm getting this on kucalk buildbot
make[3]: Entering directory '/var/lib/buildbot/slave/sage_git/build/build/make'
cd '/var/lib/buildbot/slave/sage_git/build' && source '/var/lib/buildbot/slave/sage_git/build/src/bin/sage-env' && source '/var/lib/buildbot/slave/sage_git/build/build/bin/sage-build-env-config' && sage-logger -p '/var/lib/buildbot/slave/sage_git/build/build/pkgs/sagelib/spkg-install' '/var/lib/buildbot/slave/sage_git/build/logs/pkgs/sagelib-9.1.rc5.log'
[sagelib-9.1.rc5] /var/lib/buildbot/slave/sage_git/build/local/bin/pkg-config: line 16: /doesnotexist/bin/pkgconf: No such file or directory
[sagelib-9.1.rc5] ************************************************************************
[sagelib-9.1.rc5] Traceback (most recent call last):
[sagelib-9.1.rc5] File "setup.py", line 72, in <module>
[sagelib-9.1.rc5] from module_list import ext_modules, library_order
[sagelib-9.1.rc5] File "/var/lib/buildbot/slave/sage_git/build/src/module_list.py", line 14, in <module>
[sagelib-9.1.rc5] cblas_pc = pkgconfig.parse('cblas')
[sagelib-9.1.rc5] File "/var/lib/buildbot/slave/sage_git/build/local/lib/python3.7/site-packages/pkgconfig/pkgconfig.py", line 248, in parse
[sagelib-9.1.rc5] _raise_if_not_exists(package)
[sagelib-9.1.rc5] File "/var/lib/buildbot/slave/sage_git/build/local/lib/python3.7/site-packages/pkgconfig/pkgconfig.py", line 103, in _raise_if_not_exists
[sagelib-9.1.rc5] raise PackageNotFoundError(package)
[sagelib-9.1.rc5] pkgconfig.pkgconfig.PackageNotFoundError: cblas not found
[sagelib-9.1.rc5] ************************************************************************
[sagelib-9.1.rc5] Error building the Sage library
[sagelib-9.1.rc5] ************************************************************************
[sagelib-9.1.rc5] Please email sage-devel (http://groups.google.com/group/sage-devel)
[sagelib-9.1.rc5] explaining the problem and including the relevant part of the log file
[sagelib-9.1.rc5] /var/lib/buildbot/slave/sage_git/build/logs/pkgs/sagelib-9.2.beta0.log
[sagelib-9.1.rc5] Describe your computer, operating system, etc.
[sagelib-9.1.rc5] ************************************************************************
[sagelib-9.1.rc5]
[sagelib-9.1.rc5] real 0m0.117s
[sagelib-9.1.rc5] user 0m0.100s
[sagelib-9.1.rc5] sys 0m0.012s
Branch pushed to git repo; I updated commit sha1. New commits:
1cfed7c | build/pkgs/sagelib/spkg-install: Do not poison SAGE_LOCAL, used in script installed by spkg pkgconf |
(simplification in follow-up ticket: #29779 pkg-config installed from SPKG pkgconf should not depend on environment variable SAGE_LOCAL)
kucalk buildbot -> I read on sage-devel that people suspect it to be running with
--with-python=2, which of course is meaningless with Sage 9.2+.
Perhaps we should make --with-python=2 trigger an error?
Replying to @dimpase:
kucalk buildbot-> I read on sage-devel that people suspect it to be running with
--with-python=2, which of course is meaningless with Sage 9.2+.
Perhaps we should make--with-python=2trigger an error?
#29669 makes it an error already.
But the above is unrelated to python 2 vs python 3. This error showed up on systems where no system pkg-config is available. Fixed by 1cfed7c.
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
305d8cf | Trac #29345: replace a bash array with something portable. |
d0dff56 | Trac #29345: replace a few uses of "source" with "." |
5ac420b | Trac #29345: fix some bashisms in sage-env's resolvelinks() function. |
0a61795 | Trac #29345: don't force SHELL=bash any longer. |
5db5318 | Trac #29345: remove "break" statements from AC_SEARCH_LIBS. |
e810ad1 | Trac #29345: don't use sage's config.status for the lrcalc build. |
93c9921 | Trac #29345: replace the function that populates the CVXOPT_* variables. |
0e66a0a | Trac #29345: add Dima's SPKG patches for ksh compatibility. |
df3f05e | build/make/Makefile.in [SCRIPT_PACKAGE_templ]: cd into the SPKG directory; adjust spkg-install scripts |
5372065 | Merge branch 't/29793/script_packages_should_cd_into_the_spkg_directory' into t/29411/make_sagelib_a_script_package |
Branch pushed to git repo; I updated commit sha1. New commits:
cc30471 | build/bin/write-dockerfile.sh: Do not ADD removed file src/Makefile.in |
Ready for review...
why do I see [sagelib-9.1.rc5] while building, even though
$ cat VERSION.txt
SageMath version 9.2.beta0, Release Date: 2020-05-28
When this ticket is merged, src/bin/sage-update-version will update build/pkgs/sagelib/package-version.txt, where this comes from
OK
Thanks!
rebased over rebased #29793
Last 10 new commits:
aa206bf | src/Makefile.in: Merge into build/make/Makefile.in, build/pkgs/sagelib/spkg-install |
e052741 | build/make/Makefile.in, build/pkgs/sagelib/dependencies: Make sure sagelib spkg-install is always run |
1e19861 | build/pkgs/sagelib/spkg-install: Update references to tickets |
d6ee516 | build/pkgs/sagelib/spkg-install: Also poison environment variables SAGE_LOCAL etc. |
8a4b60d | build/pkgs/sagelib/spkg-install: Add comment regarding SAGE_SPKG_INST |
4071eeb | build/pkgs/sagelib/spkg-install: Also poison SAGE_PKGCONFIG, SAGE_SPKG_SCRIPTS |
1707161 | src/setup.py: Update comment (not specific to gcc) |
985bfca | build/pkgs/sagelib/spkg-install: Do not poison SAGE_LOCAL, used in script installed by spkg pkgconf |
b212ce8 | build/pkgs/sagelib/spkg-install: Fix up error exits |
5790687 | build/bin/write-dockerfile.sh: Do not ADD removed file src/Makefile.in |
Changed branch from u/mkoeppe/make_sagelib_a_script_package to public/make_sagelib_a_script_package
Changed branch from public/make_sagelib_a_script_package to u/mkoeppe/make_sagelib_a_script_package
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:
539c182 | build/make/install: Do not depend on src/bin/sage-version.sh |
Pushed to wrong ticket, sorry
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. New commits:
Changed branch from u/mkoeppe/make_sagelib_a_script_package to cc30471
this has caused a breakage of jupyter notebook/server. Apparently some of its dependencies get "blessed"
by ./sage -b, but no longer get blessed by make build.
Running 'make build' on that ticket's commit in the develop branch (0bcd001 make sagelib a script package) produces the error for me whereas doing so on the immediately previous commit (10fb624 Trac #29289: Remove support for installing old-style SPKGs, deprecated in Sage 6.9) does not. Interestingly, it's been sort of hit-and-miss on whether the error shows up when doing just a './sage -b'.
On Saturday, July 11, 2020 at 5:26:02 PM UTC-7, Paul Masson wrote:
Just built a ticket based on 9.2.beta3 and the same problem occurs.
On Saturday, July 11, 2020 at 4:09:56 PM UTC-7, Paul Masson wrote:
After a successful incremental build from 9.2.beta2, trying to start a new notebook results in the browser console error "Failed to load resource: the server responded with a status of 404 (Not Found)" when trying to fetch Mathjax from http://localhost:8888/nbextensions/mathjax/MathJax.js. Has anyone other than Dima seen this also?
This appears to be a serious server configuration error. I don't see any ticket in either this beta or the previous one that would cause this. Am I missing some new configuration step? Sage built and runs just fine.
This broke sage -b: it wants to run make in a directory with no Makefile.