ci-cygwin.yml: Multi-stage build
mkoeppe opened this issue · 63 comments
The GitHub workflow for cygwin does not fit into the 6-hour time limit.
We do a multi-stage build, saving and loading local/ to/from build artifacts.
Depends on #29417
Depends on #29392
CC: @dimpase @embray @slel @jpflori @tscrim @vbraun @kcrisman
Component: porting: Cygwin
Author: Matthias Koeppe
Branch/Commit: 27ddea3
Reviewer: Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/29403
Commit: 79bee2f
Branch pushed to git repo; I updated commit sha1. New commits:
79bee2f | ci-cygwin.yml: Remove sage-spkg -y -o |
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
cd0a0ec | handle symlinks |
5a387d0 | Use winsymlinks:native |
122f972 | Full targets |
424d86e | prepare the final logs artifact |
f646274 | store local/var/tmp/sage/build in logs artifact |
6924328 | stage 1 |
97d3f86 | touch installation records to get same mtime |
5939bc7 | Upload log artifact first |
59d20e0 | stage 4 |
cef0d73 | more stages |
Sample run https://github.com/mkoeppe/sage/actions/runs/63708828
Something is still triggering rebuilds of targets that have been built in a previous stage.
I'd suspect it's the toolchain crud.
I'm running one test with MAKE="make --debug=verbose" to see what's happening there
Branch pushed to git repo; I updated commit sha1. New commits:
de50cf8 | better touch |
Author: Matthias Koeppe
This works pretty well now.
New failure mode: https://github.com/mkoeppe/sage/runs/560884908
"/cygdrive/d/a/sage/sage/local/lib/python3.7/subprocess.py", line 775, in __init__
[backports_abc-0.5] restore_signals, start_new_session)
[backports_abc-0.5] File "/cygdrive/d/a/sage/sage/local/lib/python3.7/subprocess.py", line 1453, in _execute_child
[backports_abc-0.5] restore_signals, start_new_session, preexec_fn)
[backports_abc-0.5] BlockingIOError: [Errno 11] Resource temporarily unavailable
[backports_abc-0.5] Cleaning up...
https://github.com/mkoeppe/sage/runs/560884917
Installing backports_abc-0.5
0 [main] python3.7 16471 child_info_fork::abort: address space needed by 'cyglzma-5.dll' (0x400000) is already occupied
Installing package backports_abc using pip3
0 [main] python3.7 16554 child_info_fork::abort: address space needed by 'cyglzma-5.dll' (0x400000) is already occupied
Ignoring indexes: https://pypi.org/simple
https://github.com/mkoeppe/sage/runs/560884917
[r-3.6.2.p0] gcc -I../../src/extra -I../../src/extra/xdr -I. -I../../src/include -I../../src/include -I/cygdrive/d/a/sage/sage/local/include -I../../src/nmath -DHAVE_CONFIG_H -fopenmp -g -O2 -fPIC -c version.c -o version.o
[r-3.6.2.p0] sysutils.c:554:10: fatal error: iconv.h: No such file or directory
[r-3.6.2.p0] 554 | #include <iconv.h>
[r-3.6.2.p0] | ^~~~~~~~~
[r-3.6.2.p0] compilation terminated.
Replying to @mkoeppe:
New failure mode: https://github.com/mkoeppe/sage/runs/560884908
"/cygdrive/d/a/sage/sage/local/lib/python3.7/subprocess.py", line 775, in __init__ [backports_abc-0.5] restore_signals, start_new_session) [backports_abc-0.5] File "/cygdrive/d/a/sage/sage/local/lib/python3.7/subprocess.py", line 1453, in _execute_child [backports_abc-0.5] restore_signals, start_new_session, preexec_fn) [backports_abc-0.5] BlockingIOError: [Errno 11] Resource temporarily unavailable [backports_abc-0.5] Cleaning up...
welcome to Cygwin rebase Hell.
One might need to call one of these src/bin/sage-rebase* (sacrificing a black chicken might be a pre-req).
In https://github.com/mkoeppe/sage/runs/572730280 (cygwin-stage-ii-a (minimal)), despite rebasing in the "Extract sage-local artifact" step, I am still getting _fork::abort: address space needed by 'cyglzma-5.dll' (0x400000) is already occupied when building packages
cygwin-stage-ii-b (minimal) times out - https://github.com/mkoeppe/sage/runs/572730298 - may need to rebalance stages
In https://github.com/mkoeppe/sage/runs/572730287 (cygwin-stage-ii-a (standard)): matplotlib fails to build - numpy-related
File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/matrixlib/__init__.py", line 6, in <module>
from .defmatrix import *
File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/matrixlib/defmatrix.py", line 13, in <module>
from numpy.linalg import matrix_power
File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/linalg/__init__.py", line 51, in <module>
from .linalg import *
File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/linalg/linalg.py", line 35, in <module>
from numpy.linalg import lapack_lite, _umath_linalg
ImportError: No such file or directory
Anyone out there who is interested in cygwin?
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
4d2366b | tox fixup |
828394b | .github/workflows/scan-logs.sh: Fixup regex |
a39ca99 | .github/workflows/scan-logs.sh: Rewording for better display |
0a02589 | tox.ini: Store successfully built docker tags in {toxenv}/Dockertags |
d209e8e | tox.yml: Retrieve logs from successful builds via {toxenv}/Dockertags |
f1858e7 | .github/workflows/tox.yml: Pass TARGETS to tox so that errors in doc-html count as failures |
f9aca59 | .github/workflows/scan-logs.sh: Fixup regexp for scanning ptest.log |
b32e1bf | .github/workflows/scan-logs.sh: Also catch errors of the form ImportError |
d94969c | tox.ini: Quoting fix |
c08176e | Merge branch 't/29417/tox-ci-minor-fixes' into t/29403/ci_cygwin_yml__multi_stage_build |
As far as I can tell, this is okay, but I am not enough of an expert to say definitively. I would appreciate a second opinion from someone who is more well-versed in the Sage build system to double-check.
The issue of comment 21 is not resolved yet
Ancient info on rebasing: https://groups.google.com/d/msg/sage-windows/ygK1kJm9p9w/x0ckGqo2ej4J
well, yes, it was not clear whether rebasing must be done from outside of Cygwin shell, or not.
I gather that we rebase within Cygwin.
By the way, it would be interesting to do testing in Windows WSL...
I'll give it another shot, doing exactly what is also done in sage-spkg
Branch pushed to git repo; I updated commit sha1. New commits:
faf0ed6 | tox.ini: Another quoting fix |
aed3fb8 | src/doc/en/developer/portability_testing.rst: Explain how to read [GitHub](../wiki/GitHub) Actions page |
d39d627 | tox.yml [python3_xcode]: Install the package |
ba0c266 | fixup |
9060d23 | fixup |
b7bb418 | tox.yml [macos-python3_pythonorg]: Use sudo with installer |
4b19fdd | Merge branch 't/29417/tox-ci-minor-fixes' into t/29403/ci_cygwin_yml__multi_stage_build |
55c8a48 | tox.yml: Use rebase.sh |
ffdc97e | fixup merge |
with these multi-stage builds, where *.so get built at one place, and moved then somewhere esle, it could be that more rebasing is needed. In particular, I recall seeing and fixing rebase locations for Python modules being built.
(Basically it was the issue that totally broke building Sage on Cygwin32, simply due to insufficient address space).
Testing at https://github.com/mkoeppe/sage/actions/runs/80516745
Yes, I'm rebasing after unpacking local from the previous stages.
I think it's working now, and we can see the real build errors again in cygwin-standard: matplotlib, rpy2, scipy (likely BLAS-related) https://github.com/mkoeppe/sage/runs/596536457
Branch pushed to git repo; I updated commit sha1. New commits:
fc61002 | ci-cygwin.yml: Package up SAGE_LOCAL/var/tmp/sage/build to avoid error with actions/upload-artifact |
testing on https://github.com/dimpase/sage/actions/runs/81294070
I cannot reproduce the matplotlib build error locally - even when I download and unpack stage-i-a and stage-i-b (standard) and configure against that.
Branch pushed to git repo; I updated commit sha1. New commits:
59ec460 | ./configure --enable-experimental-packages --enable-download-from-upstream-url |
c9cf3cc | Merge branch 't/29392/__configure___enable_experimental_packages___enable_download_from_upstream_url' into t/29403/ci_cygwin_yml__multi_stage_build |
037f508 | tox.yml: Pass CONFIGURE_ARGS: --enable-experimental-packages --enable-download-from-upstream-url |
Branch pushed to git repo; I updated commit sha1. New commits:
998f02d | ci-cygwin.yml: Fixup for: Package up SAGE_LOCAL/var/tmp/sage/build |
Branch pushed to git repo; I updated commit sha1. New commits:
d6bf4b1 | ci-cygwin.yml: Install standard system packages also for 'stndard-python2' |
how does one need to modify ~/.github/ to make GH Actions only run cygwin tests?
Replying to @dimpase:
how does one need to modify ~/.github/ to make GH Actions only run cygwin tests?
Just delete the file tox.yml
Branch pushed to git repo; I updated commit sha1. New commits:
2779e24 | ci-cygwin.yml: Another fixup for: Package up SAGE_LOCAL/var/tmp/sage/build |
Branch pushed to git repo; I updated commit sha1. New commits:
27ddea3 | build/bin/sage-logger: Back out an experimental change |
Reviewer: Dima Pasechnik
Thanks!
Changed branch from u/mkoeppe/ci_cygwin_yml__multi_stage_build to 27ddea3