sage_bootstrap: Update/extend "sage -package", fix "make download"

Question

sage_bootstrap: Update/extend "sage -package", fix "make download"

mkoeppe opened this issue 5 years ago · 56 comments

There are several places where the spkg (build/pkgs/*/) and system package information (build/pkgs/*/distros/*.txt) is parsed:

bootstrap
src/doc/bootstrap
build/bin/write_dockerfile.sh
build/bin/sage-get-system-packages
tox.ini (for local-homebrew, local-conda)
.github/workflows/ci-wsl.yml (using powershell)
src/bin/sage-download-upstream
m4/sage_spkg_*.m4

The purpose of this ticket is to prepare refactoring of this code by extending the sage_bootstrap package.

We update sage_bootstrap.package so that it understands non-normal packages (i.e., script and pip packages, which do not have checksums.ini)
We change sage -package list so it supports the following invocations:
- sage -package list :standard: :optional: (several "classes" (types), not just one)
- sage -package list --has-file=spkg-configure.m4 :standard: (needed for bootstrap, src/doc/bootstrap)
- sage -package list --has-file=spkg-configure.m4 --has-file=distros/debian.txt :standard: :optional: (needed for build/bin/write-dockerfile.sh, tox.ini)
- sage -package list --has-file=SPKG.rst :all: (needed for src/doc/bootstrap)
We fix and extend sage -package download (follow up from #30648) so it supports
- sage -package download :all: (again - was broken)
- sage -package download --allow-upstream :all: (allows retrieving from upstream_url)
  and use it for make download (#29896), removing src/bin/sage-download-upstream

To run the testsuite of sage_bootstrap, use

  (cd build && tox)

Follow-up:

The actual refactoring of the above scripts (#30947, #30951, #30968, ...)
Wishlist item: sage -package list --not-source=pip :optional: :experimental: (would simplify bootstrap)

Depends on #30648

CC: @seblabbe @tobiasdiez @slel @jhpalmieri @vbraun

Component: build: configure

Author: Matthias Koeppe

Branch: f33c363

Reviewer: Sébastien Labbé

Issue created by migration from https://trac.sagemath.org/ticket/30865

Answer 1 · 2020-11-05T18:27:05.000Z

comment:1

Developers interested in improving the system package advice may want to work on this ticket

Answer 2 · 2020-11-19T17:58:45.000Z

Author: Matthias Koeppe

Answer 3 · 2020-11-19T17:58:45.000Z

Description changed:

--- 
+++ 
@@ -6,7 +6,12 @@
 - `tox.ini` (for `local-homebrew`, `local-conda`)
 - `.github/workflows/ci-wsl.yml` (using powershell)
 
-The purpose of this ticket is to refactor it by reimplementing it as a new Python module of the `sage_bootstrap` package.
+The purpose of this ticket is to refactor it by extending the `sage_bootstrap` package.
 
 
+To run the testsuite of `sage_bootstrap`, use
 
+```
+  (cd build && tox)
+```
+

Answer 4 · 2020-11-19T18:04:45.000Z

Branch: u/mkoeppe/sage_bootstrap__update_extend_system_package_tools

Answer 5 · 2020-11-19T18:45:39.000Z

New commits:

`2265048`	`sage_bootstrap.package: Handle non-normal packages (which have no checksums.ini etc.)`
`c405d10`	`sage_bootstrap.package.Package._init_type: Do not accept 'script' or 'pip' any more`

Answer 6 · 2020-11-19T18:45:39.000Z

Commit: c405d10

Answer 7 · 2020-11-19T18:45:39.000Z

Description changed:

--- 
+++ 
@@ -7,7 +7,13 @@
 - `.github/workflows/ci-wsl.yml` (using powershell)
 
 The purpose of this ticket is to refactor it by extending the `sage_bootstrap` package.
-
+- We update `sage_bootstrap.package` so that it understands non-normal packages (i.e., script and pip packages, which do not have `checksums.ini`)
+- We change `sage -package list` so it supports the following invocations:
+  - `sage -package list :standard: :optional:` (several "classes" (types), not just one)
+  - `sage -package list --has-file=spkg-configure.m4 :standard:` (needed for `bootstrap`, `src/doc/bootstrap`)
+  - `sage -package list --has-file=spkg-configure.m4 --has-file=distros/debian.txt :standard: :optional:` (needed for `build/bin/write-dockerfile.sh`, `tox.ini`)
+  - `sage -package list --has-file=SPKG.rst :all:` (needed for `src/doc/bootstrap`)
+  - `sage -package list --not-source=pip :optional: :experimental:` (needed for `bootstrap`)
 
 To run the testsuite of `sage_bootstrap`, use

Answer 8 · 2020-11-19T18:56:52.000Z

Dependencies: #30648

Answer 9 · 2020-11-19T19:02:11.000Z

Description changed:

--- 
+++ 
@@ -14,6 +14,8 @@
   - `sage -package list --has-file=spkg-configure.m4 --has-file=distros/debian.txt :standard: :optional:` (needed for `build/bin/write-dockerfile.sh`, `tox.ini`)
   - `sage -package list --has-file=SPKG.rst :all:` (needed for `src/doc/bootstrap`)
   - `sage -package list --not-source=pip :optional: :experimental:` (needed for `bootstrap`)
+- We fix and extend `sage -package download` likewise (follow up from #30648) so it (again) supports
+  - `sage -package download :all:`
 
 To run the testsuite of `sage_bootstrap`, use

Answer 10 · 2020-11-19T19:07:25.000Z

Description changed:

--- 
+++ 
@@ -14,8 +14,9 @@
   - `sage -package list --has-file=spkg-configure.m4 --has-file=distros/debian.txt :standard: :optional:` (needed for `build/bin/write-dockerfile.sh`, `tox.ini`)
   - `sage -package list --has-file=SPKG.rst :all:` (needed for `src/doc/bootstrap`)
   - `sage -package list --not-source=pip :optional: :experimental:` (needed for `bootstrap`)
-- We fix and extend `sage -package download` likewise (follow up from #30648) so it (again) supports
-  - `sage -package download :all:`
+- We fix and extend `sage -package download` likewise (follow up from #30648) so it supports
+  - `sage -package download :all:` (again - was broken)
+  - `sage -package download --allow-upstream :all:` (allows retrieving from `upstream_url`)
 
 To run the testsuite of `sage_bootstrap`, use

Answer 11 · 2020-11-19T20:12:02.000Z

Changed commit from c405d10 to 3f91313

Answer 12 · 2020-11-19T20:12:02.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`e463513`	`30648: download_cls -> download`
`0b343d1`	`Merge commit 'e4635137337490ce452945728c8667f3f51b5b40' of git://trac.sagemath.org/sage into t/30865/sage_bootstrap__update_extend_system_package_tools`
`5fc8441`	`sage_bootstrap.app.Application.download_cls: Pass allow_upstream to download`
`794a15c`	`sage_bootstrap.cndline: Back to using download_cls`
`6369bb4`	`sage -package list: Handle --has-file, multiple package classes (types)`
`4fbf753`	`src/doc/en/installation/source.rst: Remove outdated documentation on old-style packages`
`7082715`	`Makefile: Update documentation of 'make download'`
`3f91313`	`src/bin/sage-download-upstream: Remove, use 'sage --package download' instead`

Answer 13 · 2020-11-19T20:42:24.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`69f7ea8`	`sage_bootstrap.PackageClass: Accept several arguments, simplify use`
`40d6604`	`sage_bootstrap.package.Package.has_file: New, use it`
`6475263`	`sage_bootstrap.PackageClass: Handle filter has_files, simplify use`
`769f635`	`sage_bootstrap.tarball: Raise an error if attempting to download a tarball of a non-normal package`
`f6b79bb`	`sage_bootstrap.app.download_cls: Only attempt to download tarballs for normal packages (with checksums.ini)`

Answer 14 · 2020-11-19T20:42:24.000Z

Changed commit from 3f91313 to f6b79bb

Answer 15 · 2020-11-19T20:43:11.000Z

Description changed:

--- 
+++ 
@@ -13,10 +13,10 @@
   - `sage -package list --has-file=spkg-configure.m4 :standard:` (needed for `bootstrap`, `src/doc/bootstrap`)
   - `sage -package list --has-file=spkg-configure.m4 --has-file=distros/debian.txt :standard: :optional:` (needed for `build/bin/write-dockerfile.sh`, `tox.ini`)
   - `sage -package list --has-file=SPKG.rst :all:` (needed for `src/doc/bootstrap`)
-  - `sage -package list --not-source=pip :optional: :experimental:` (needed for `bootstrap`)
-- We fix and extend `sage -package download` likewise (follow up from #30648) so it supports
+- We fix and extend `sage -package download` (follow up from #30648) so it supports
   - `sage -package download :all:` (again - was broken)
   - `sage -package download --allow-upstream :all:` (allows retrieving from `upstream_url`)
+  and use it for `make download` (#29896).
 
 To run the testsuite of `sage_bootstrap`, use
 
@@ -24,3 +24,6 @@
   (cd build && tox)
 ```
 
+Follow-up wishlist item:
+- `sage -package list --not-source=pip :optional: :experimental:` (would simplify `bootstrap`)
+

Answer 16 · 2020-11-19T20:47:20.000Z

Description changed:

--- 
+++ 
@@ -1,10 +1,14 @@
 (from #29146, #30861)
 
 There are several places where the system package information (`build/pkgs/*/distros/*.txt`) is parsed:
+- `bootstrap`
+- `src/doc/bootstrap`
 - `build/bin/write_dockerfile.sh`
 - `build/bin/sage-get-system-packages`
 - `tox.ini` (for `local-homebrew`, `local-conda`)
 - `.github/workflows/ci-wsl.yml` (using powershell)
+- `src/bin/sage-download-upstream`
+- `m4/sage_spkg_*.m4`
 
 The purpose of this ticket is to refactor it by extending the `sage_bootstrap` package.
 - We update `sage_bootstrap.package` so that it understands non-normal packages (i.e., script and pip packages, which do not have `checksums.ini`)
@@ -16,7 +20,7 @@
 - We fix and extend `sage -package download` (follow up from #30648) so it supports
   - `sage -package download :all:` (again - was broken)
   - `sage -package download --allow-upstream :all:` (allows retrieving from `upstream_url`)
-  and use it for `make download` (#29896).
+  and use it for `make download` (#29896), removing `src/bin/sage-download-upstream`
 
 To run the testsuite of `sage_bootstrap`, use

Answer 17 · 2020-11-19T20:54:34.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4be654b`	`sage_bootstrap.cmdline: Initialize has_files filter correctly`

Answer 18 · 2020-11-19T20:54:34.000Z

Changed commit from f6b79bb to 4be654b

Answer 19 · 2020-11-19T20:56:56.000Z

comment:13

Ready for review

Answer 20 · 2020-11-19T20:56:56.000Z

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 - `src/bin/sage-download-upstream`
 - `m4/sage_spkg_*.m4`
 
-The purpose of this ticket is to refactor it by extending the `sage_bootstrap` package.
+The purpose of this ticket is to prepare refactoring of this code by extending the `sage_bootstrap` package.
 - We update `sage_bootstrap.package` so that it understands non-normal packages (i.e., script and pip packages, which do not have `checksums.ini`)
 - We change `sage -package list` so it supports the following invocations:
   - `sage -package list :standard: :optional:` (several "classes" (types), not just one)
@@ -28,6 +28,10 @@
   (cd build && tox)
 ```
 
-Follow-up wishlist item:
-- `sage -package list --not-source=pip :optional: :experimental:` (would simplify `bootstrap`)
 
+
+Follow-up:
+- The actual refactoring of the above scripts
+- Wishlist item: `sage -package list --not-source=pip :optional: :experimental:` (would simplify `bootstrap`)
+
+

Answer 21 · 2020-11-19T20:57:48.000Z

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
 (from #29146, #30861)
 
-There are several places where the system package information (`build/pkgs/*/distros/*.txt`) is parsed:
+There are several places where the spkg (`build/pkgs/*/`) and system package information (`build/pkgs/*/distros/*.txt`) is parsed:
 - `bootstrap`
 - `src/doc/bootstrap`
 - `build/bin/write_dockerfile.sh`

Answer 22 · 2020-11-22T09:18:49.000Z

comment:16

The current branch is doing this:

-    def download_cls(self, package_name_or_class):
+    def download_cls(self, package_name_or_class, allow_upstream=False):

So should the change I made in #30648 be reverted then?

(I recall that the problem fixed in #30648 was that download_cls method had no argument allow_upstream where download method did, so I changed download_cls to download. But at the time, I was wondering if that was the good fix. Because another fix could have been to add the allow_upstream argument to the download_cls method which you are doing in this branch)

Answer 23 · 2020-11-22T09:49:33.000Z

comment:17

Doing += on lists creates a new list in memory as opposed to append or extend:

sage: L = list(range(10))                                                       
sage: K = list('abcd')                                                          
sage: id(L)                                                                     
140556349258880
sage: id(L+K)                                                                   
140556349147776
sage: L.extend(K)                                                               
sage: id(L)                                                                     
140556349258880

So using += to append elements to a list uses a quadratic amount of memory until the garbage collector gets called.

So I would suggest the following change:

- self.names += [package_name_or_class]
+ self.names.append(package_name_or_class)

as well as (5 times):

- self.names += [pkg.name for pkg in Package.all() if predicate(pkg)]
+ self.names.extend(pkg.name for pkg in Package.all() if predicate(pkg))

Answer 24 · 2020-11-22T09:50:57.000Z

comment:18

I will try the branch later today, and come back with further comments if I have.

Answer 25 · 2020-11-22T20:25:08.000Z

comment:19

Replying to @seblabbe:

The current branch is doing this:
-    def download_cls(self, package_name_or_class):
+    def download_cls(self, package_name_or_class, allow_upstream=False):
So should the change I made in #30648 be reverted then?

Yes, in fact I have already reverted it as part of the branch on this ticket.

Answer 26 · 2020-11-22T20:28:52.000Z

Changed commit from 4be654b to b762c3d

Answer 27 · 2020-11-22T20:28:52.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b762c3d`	`sage_bootstrap.expand_class: Use .append, .extend instead of += for lists`

Answer 28 · 2020-11-22T20:58:15.000Z

Description changed:

--- 
+++ 
@@ -31,7 +31,7 @@
 
 
 Follow-up:
-- The actual refactoring of the above scripts
+- The actual refactoring of the above scripts (#30947, ...)
 - Wishlist item: `sage -package list --not-source=pip :optional: :experimental:` (would simplify `bootstrap`)

Answer 29 · 2020-11-23T02:29:27.000Z

Description changed:

--- 
+++ 
@@ -31,7 +31,7 @@
 
 
 Follow-up:
-- The actual refactoring of the above scripts (#30947, ...)
+- The actual refactoring of the above scripts (#30947, #30951, ...)
 - Wishlist item: `sage -package list --not-source=pip :optional: :experimental:` (would simplify `bootstrap`)

Answer 30 · 2020-11-23T08:35:23.000Z

comment:23

Replying to @mkoeppe:

Yes, in fact I have already reverted it as part of the branch on this ticket.

Ok, yes, I agree. Since #30648 is not in sage yet, this is why I don't see it in the diff.

Answer 31 · 2020-11-23T09:51:58.000Z

comment:24

(cd build && tox)

I did that. It finishes with a big list of typos and says that there were errors while running the tests. When I scroll back, I can't see the errors, because the typos takes all of the lines in the finite history of the terminal. Are the output of tox logged somewhere?

I find it weird to call it class of package and use cls (which is often use for representing a Python class. Maybe nature of the package? Anyway, this is out of topic.

(0) I confirm that sage --package list works with inputs such as
--has-file=spkg-configure.m4, --has-file=distros/debian.txt, --has-file=SPKG.rst. Also sage --package download works. Also make download works.

Sorry, I still have few more comments listed below.

(1) When I read the documentation of sage --package, I don't know what Sage Bootstrap Library means. I suggest to add a sentence in the file sage_bootstrap.cmdline.py to explain it. Something as follows, but please replace the verb deal with any more precise verb:

 description = \
 """
 Sage Bootstrap Library
+
+It deals with all of the packages in SageMath.
 """

(2) I don't like the following error message:

$ sage --package list :standardd:
Traceback (most recent call last):
  File "/home/slabbe/GitBox/sage/build/bin/sage-package", line 42, in <module>
    run()
  File "/home/slabbe/GitBox/sage/build/bin/../sage_bootstrap/cmdline.py", line 306, in run
    app.list_cls(*args.package_class, has_files=args.has_files)
  File "/home/slabbe/GitBox/sage/build/bin/../sage_bootstrap/app.py", line 69, in list_cls
    pc = PackageClass(*package_classes, **filters)
  File "/home/slabbe/GitBox/sage/build/bin/../sage_bootstrap/expand_class.py", line 48, in __init__
    raise ValueError('Package name cannot start with ":", got %s', package_name_or_class)
ValueError: ('Package name cannot start with ":", got %s', ':standardd:')

It tells me the input cannot start with ":", but this is false, as :standard: is accepted.

Thus, I would suggest the following change in sage_bootstrap/expand_class.py:

             else:
-                if package_name_or_class.startswith(':'):
-                    raise ValueError('Package name cannot start with ":", got %s', package_name_or_class)
-                if package_name_or_class.endswith(':'):
-                    raise ValueError('Package name cannot end with ":", got %s', package_name_or_class)
+                if package_name_or_class.startswith(':') or package_name_or_class.endswith(':'):
+                    raise ValueError('Valid package classes are :all:, :standard:, :optional:, :experimental: or :huge:, got %s', package_name_or_class)
                 self.names.append(package_name_or_class)

Question: do you confirm that the following is a desirable feature?

$ sage --package list pari
pari

(3) Finally, in build/sage_bootstrap/package.py, I suggest:

-        if pattern:
-            return self.tarball_pattern.replace('VERSION', self.version)
+        if pattern:
+            return pattern.replace('VERSION', self.version)

(4) Question: What is the meaning of the expression non-normal in the file build/sage_bootstrap/tarball.py ?

I am currently waiting for the report of make ptestlong.

Answer 32 · 2020-11-23T09:59:21.000Z

comment:25

The make ptestlong finished with:

----------------------------------------------------------------------
sage -t --long --random-seed=0 src/sage/interfaces/singular.py  # Killed due to segmentation fault
----------------------------------------------------------------------

which is unrelated to this ticket.

Answer 33 · 2020-11-23T11:54:51.000Z

comment:26

Replying to @seblabbe:

The make ptestlong finished with:

----------------------------------------------------------------------
sage -t --long --random-seed=0 src/sage/interfaces/singular.py  # Killed due to segmentation fault
----------------------------------------------------------------------

which is unrelated to this ticket.

Indeed that is different, and tracked at #30945.

Answer 34 · 2020-11-23T14:41:59.000Z

Reviewer: Sébastien Labbé

Answer 35 · 2020-11-23T18:29:23.000Z

comment:28

Replying to @seblabbe:

(cd build && tox)

I did that. It finishes with a big list of typos

If it shows typos, then you probably ran sage -tox, not tox. If you don't have tox in your global environment, you can use (cd build && sage -sh -c tox).

Are the output of tox logged somewhere?

Yes, it creates a subdirectory .tox, where you'll find logs.

Answer 36 · 2020-11-23T19:01:53.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`51db2a9`	`sage_bootstrap: Eliminate terminology 'package class' in help and error messages`

Answer 37 · 2020-11-23T19:01:53.000Z

Changed commit from b762c3d to 51db2a9

Answer 38 · 2020-11-23T19:04:00.000Z

comment:30

Replying to @seblabbe:

I find it weird to call it class of package and use cls (which is often use for representing a Python class. Maybe nature of the package? Anyway, this is out of topic.

I agree. I have now made a change to eliminate this (unexplained) terminology from user-facing messages. But I haven't changed the class name PackageClass or argument names, to keep the diff small.

Answer 39 · 2020-11-23T19:05:33.000Z

comment:31

Replying to @seblabbe:

(4) Question: What is the meaning of the expression non-normal in the file build/sage_bootstrap/tarball.py ?

This is referring to https://doc.sagemath.org/html/en/developer/packaging.html#package-source-types

Answer 40 · 2020-11-23T19:07:56.000Z

comment:32

Replying to @seblabbe:

I would suggest the following change in sage_bootstrap/expand_class.py:

             else:
-                if package_name_or_class.startswith(':'):
-                    raise ValueError('Package name cannot start with ":", got %s', package_name_or_class)
-                if package_name_or_class.endswith(':'):
-                    raise ValueError('Package name cannot end with ":", got %s', package_name_or_class)
+                if package_name_or_class.startswith(':') or package_name_or_class.endswith(':'):
+                    raise ValueError('Valid package classes are :all:, :standard:, :optional:, :experimental: or :huge:, got %s', package_name_or_class)
                 self.names.append(package_name_or_class)

I have made a similar change in the above commit.

Question: do you confirm that the following is a desirable feature?
$ sage --package list pari
pari

Yes, this is a feature, and the above commit documents it.

Answer 41 · 2020-11-23T19:19:36.000Z

Changed commit from 51db2a9 to c0a0329

Answer 42 · 2020-11-23T19:19:36.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`c0a0329`	`sage_bootstrap.cmdline: Expand description`

Answer 43 · 2020-11-23T21:48:16.000Z

comment:35

Replying to @mkoeppe:

If it shows typos, then you probably ran sage -tox, not tox. If you don't have tox in your global environment, you can use (cd build && sage -sh -c tox).

Well, I first did sudo apt install tox on a Ubuntu 18.04 but, then I got problems because that tox is using python2. So, your guess is right, that was after the run of sage -tox.

Are the output of tox logged somewhere?

Yes, it creates a subdirectory .tox, where you'll find logs.

Great. Thank you.

Answer 44 · 2020-11-23T21:53:19.000Z

comment:36

Thanks for the answers and commits, I will check this tomorrow morning Bordeaux time.

[I think my comment (3) is still not fixed (well, it is just a small detail).]

Answer 45 · 2020-11-23T23:25:48.000Z

comment:37

Replying to @seblabbe:

I first did sudo apt install tox on a Ubuntu 18.04 but, then I got problems because that tox is using python2.

That should actually work too. sage_bootstrap is designed to work on a wide range of Python versions including 2.x. tox provisions virtual environments to test the package with each available Python version. To run it with a specific Python version, you can do tox -e py37 etc. It does not matter on what version of Python tox itself runs.

Answer 46 · 2020-11-23T23:28:20.000Z

Changed commit from c0a0329 to f33c363

Answer 47 · 2020-11-23T23:28:20.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f33c363`	`sage_bootstrap.package.Package.tarball_pattern: Simplify code`

Answer 48 · 2020-11-24T08:03:34.000Z

comment:39

Replying to @mkoeppe:

Replying to @seblabbe:

I first did sudo apt install tox on a Ubuntu 18.04 but, then I got problems because that tox is using python2.

That should actually work too. sage_bootstrap is designed to work on a wide range of Python versions including 2.x. tox provisions virtual environments to test the package with each available Python version. To run it with a specific Python version, you can do tox -e py37 etc. It does not matter on what version of Python tox itself runs.

Ok, I see. But after cd build, the tox command prints few lines and seems to get stuck:

$ cd build
$ tox
GLOB sdist-make: /home/slabbe/GitBox/sage/build/setup.py
py26 create: /home/slabbe/GitBox/sage/build/.tox/py26
ERROR: InterpreterNotFound: python2.6
py27 create: /home/slabbe/GitBox/sage/build/.tox/py27
py27 inst: /home/slabbe/GitBox/sage/build/.tox/dist/sage_bootstrap-1.0.zip
py27 installed: DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.,pkg-resources==0.0.0,sage-bootstrap @ file:///home/slabbe/GitBox/sage/build/.tox/dist/sage_bootstrap-1.0.zip
py27 runtests: PYTHONHASHSEED='3230691582'
py27 runtests: commands[0] | python2.7 -m unittest discover
......

Answer 49 · 2020-11-24T08:11:02.000Z

comment:40

Actually, it did not get stuck, but I obtain three errors after few minutes:

$ tox
GLOB sdist-make: /home/slabbe/GitBox/sage/build/setup.py
py26 create: /home/slabbe/GitBox/sage/build/.tox/py26
ERROR: InterpreterNotFound: python2.6
py27 create: /home/slabbe/GitBox/sage/build/.tox/py27
py27 inst: /home/slabbe/GitBox/sage/build/.tox/dist/sage_bootstrap-1.0.zip
py27 installed: DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.,pkg-resources==0.0.0,sage-bootstrap @ file:///home/slabbe/GitBox/sage/build/.tox/dist/sage_bootstrap-1.0.zip
py27 runtests: PYTHONHASHSEED='3230691582'
py27 runtests: commands[0] | python2.7 -m unittest discover
......FF..............[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
[......................................................................]
......F........ERROR [package|all:249]: Failed to open pari
..
======================================================================
FAIL: test_error (test.test_download.DownloadTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/slabbe/GitBox/sage/build/test/test_download.py", line 49, in test_error
    self.assertIsNotFoundError(log.messages())
  File "/home/slabbe/GitBox/sage/build/test/test_download.py", line 64, in assertIsNotFoundError
    "[Errno 404] Not Found: '//files.sagemath.org/sage_bootstrap/this_url_does_not_exist'"))
AssertionError: False is not true

======================================================================
FAIL: test_ignore_errors (test.test_download.DownloadTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/slabbe/GitBox/sage/build/test/test_download.py", line 57, in test_ignore_errors
    self.assertIsNotFoundError(log.messages())
  File "/home/slabbe/GitBox/sage/build/test/test_download.py", line 64, in assertIsNotFoundError
    "[Errno 404] Not Found: '//files.sagemath.org/sage_bootstrap/this_url_does_not_exist'"))
AssertionError: False is not true

======================================================================
FAIL: test_checksum (test.test_tarball.TarballTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/slabbe/GitBox/sage/build/test/test_tarball.py", line 59, in test_checksum
    '[......................................................................]\n')
AssertionError: '[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]\n[......................................................................]\n' != '[......................................................................]\n'

----------------------------------------------------------------------
Ran 39 tests in 228.835s

FAILED (failures=3)

After that it seems tox does the same with other version of Python it finds on the system. With python 3.6, it returns two errors instead of three (the test_checksum does not fail).

Let me now check whether this is related to the current ticket or not by rerunning it after a git checkout develop.

Answer 50 · 2020-11-24T08:32:01.000Z

comment:41

I also get errors with 9.3.beta1, but not exactly the same. test_ignore_errors and test_error both fails as with the branch. But test_download fails only on 9.3.beta1 whereas test_checksum fails only with the current branch.

======================================================================
FAIL: test_error (test.test_download.DownloadTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/slabbe/GitBox/sage/build/test/test_download.py", line 49, in test_error
    self.assertIsNotFoundError(log.messages())
  File "/home/slabbe/GitBox/sage/build/test/test_download.py", line 63, in assertIsNotFoundError
    self.assertTrue(messages[0][1].endswith(
AssertionError: False is not true

======================================================================
FAIL: test_ignore_errors (test.test_download.DownloadTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/slabbe/GitBox/sage/build/test/test_download.py", line 57, in test_ignore_errors
    self.assertIsNotFoundError(log.messages())
  File "/home/slabbe/GitBox/sage/build/test/test_download.py", line 63, in assertIsNotFoundError
    self.assertTrue(messages[0][1].endswith(
AssertionError: False is not true

======================================================================
FAIL: test_download (test.test_package_cmdline.SagePackageTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/slabbe/GitBox/sage/build/test/test_package_cmdline.py", line 115, in test_download
    self.assertTrue(stderr.startswith('Using cached file'))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 39 tests in 134.424s

FAILED (failures=3)

I don't know how much this is related to the current ticket. From my point of view, I give a positive review to the commits done up to now.

Matthias, I let you change the status to positive review if you think the tox issues are unrelated or if they can be dealt in another ticket.

Answer 51 · 2020-11-24T17:28:29.000Z

comment:42

Thank you!

(I also get the additional test_checksum failure on this branch. This test only passes on a release tag; this is not specific to this ticket. The other failures, also not specific to this ticket, should be investigated on a separate ticket.)

Answer 52 · 2020-11-27T04:05:46.000Z

Description changed:

--- 
+++ 
@@ -31,7 +31,7 @@
 
 
 Follow-up:
-- The actual refactoring of the above scripts (#30947, #30951, ...)
+- The actual refactoring of the above scripts (#30947, #30951, #30968, ...)
 - Wishlist item: `sage -package list --not-source=pip :optional: :experimental:` (would simplify `bootstrap`)

Answer 53 · 2020-11-29T11:57:50.000Z

Changed branch from u/mkoeppe/sage_bootstrap__update_extend_system_package_tools to f33c363

Answer 54 · 2020-12-10T02:18:07.000Z

comment:45

I tried make download and it failed to download scipoptsuite.

Which is expected. Indeed, reading from the last scipoptsuite ticket:

#24662: Upgrade scipoptsuite to 5.0.1

Upstream archive:
(DO NOT put on sage servers -- we cannot redistribute this archive)

How do we deal with that?

Answer 55 · 2020-12-10T02:18:07.000Z

Changed commit from f33c363 to none

Answer 56 · 2020-12-10T03:20:46.000Z

comment:46

Replying to @slel:

(DO NOT put on sage servers -- we cannot redistribute this archive)

How do we deal with that?

Let's discuss this on #31034