Make it possible to build on WSL
tobiasdiez opened this issue · 43 comments
This ticket is to inform you about how to build sage with the Windows Subsystem for Linux and to provide the necessary changes to make this work. Since this is my first contribution to sage, please let me know if I've overseen something obvious and please be lenient in case I broke some written or unwritten rules.
Steps to build sage on WSL
including set-up git infrastructure since this is partly different on Windows as well.
Prerequisites
Git (run everything in git bash)
-
Clone repository
-
Correct line endings (necessary if global core.autocrlf is set to true):
git config --local core.autocrlf false git rm --cached -r . git reset --hard
* Setup remote trac (nessary if you want to contribute)
Follow the instructions to set-up the ssh key https://help.github.com/en/articles/working-with-ssh-key-passphrases and then follow http://doc.sagemath.org/html/en/developer/trac.html#trac-authentication-through-ssh and http://doc.sagemath.org/html/en/developer/manual_git.html#chapter-manual-git
git remote add trac git://trac.sagemath.org/sage.git -t master
git remote set-url --push trac git@trac.sagemath.org:sage.git
* Switch to development branch
git checkout develop
## Build sage (run everything from Ubuntu bash)
* Install prerequisites
This is a slightly extended version of the [corresponding script for Linux](http://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation).
sudo apt-get update
sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev libffi-dev libgf2x-dev libcurl4-openssl-dev curl yasm
sudo apt-get install gettext libgettextpo-dev
sudo apt-get install automake pkgconf
* Change into sage folder
* Set how many cores you want to use for building (important, otherwise it will take ages)
export MAKE="make -j4"
* Build (there will be errors on the way, so ignore them using the `-k` switch)
make -k
* Test sage
./sage
2+2
# Changes necessary to make this work
In order to make the above procedure work, I had to change a few things on the way. I have pushed these changes to the branch `public/wls`.
* Linux symlinks do not work under Windows. Thus, if you clone the repository the symlinks are replaced by ordinary text files containing the target path of the symlink. To fix this, I replaced the symlinks in `build/pgks/gfortran/` by the respective files in `build/pgks/gcc` and similarily in `build/pgks/python2/` by the respective files in `build/pgks/python3/`. (There is probably a more elegant way to do this)
* Error
spgk-install should not be marked executable in the build/pkgs directory
was fixed by changing `build/bin/sage-spkg`. This is a quick hack and probably needs to be fixed properly. The trick used for cygwin does not work since `uname = Linux` when run from WSL.
* Error
OSError: [Errno 22] Invalid argument: './src/doc/pari/Programming_in_GP:_other_specific_functions.html'
open sage/upstream/giac-X.tar.bz2
was fixed by changes in `tar_file.py`. This is also a quick hack and needs a better fix, e.g. rename the file to a valid name?
* Error
The headers or library files could not be found for zlib,
a required dependency when compiling Pillow from source.
Please see the install instructions at:
https://pillow.readthedocs.io/en/latest/installation.html
was fixed by deactivating `zlib` in `pillow/spkg-install`. Note sure if this has any negative side effects.
# Remaining issues
With the above changes, the `make` build works in principle. There are a few small issues that needs further improvement.
* Creating the documentation during `make` or by `./sage --docbuild all html` fails with the errors of the form
OSError: WARNING: Unable to fetch /mnt/c/sage/local/share/doc/sage/doctrees/en/reference/algebras/environment.pickle
There is a syntax error in your configuration file: invalid syntax (conf.py, line 1)
Build finished. The built documents can be found in /mnt/c/sage/local/share/doc/sage/inventory/en/reference/algebras
I guess, the problem is that `conf.py` in `src\doc\en\reference\algebras` is a symbolic link.
* At startup of sage, you get the warning
/mnt/c/sage/local/lib/python2.7/site-packages/psutil/_pslinux.py:519: RuntimeWarning: 'sin' and 'sout' swap memory stats couldn't be determined and were set to 0 ([Errno 2] No such file or directory: '/proc/vmstat')
warnings.warn(msg, RuntimeWarning)
Probably, there is nothing we can do, since this is a known limitation of WSL: https://github.com/microsoft/WSL/issues/1071
CC: @embray @slel @dimpase @soehms @pfasante
Component: **build**
Keywords: **windows, wsl, make**
Branch/Commit: **[public/wls](https://github.com/sagemath/sagetrac-mirror/tree/public/wls) @ [`cd8116e`](https://github.com/sagemath/sagetrac-mirror/commit/cd8116ead3b66e949c414bce76e154eddf69e47c)**
Reviewer: **Tobias Diez**
_Issue created by migration from https://trac.sagemath.org/ticket/28549_
Branch pushed to git repo; I updated commit sha1. New commits:
dcb1e1b | Ignore errors that scripts are marked executable |
16b3269 | Replace symbolic links with their target |
e13d5c0 | Disable zlib for pillow |
ac782b4 | Improve error message in case checksum.ini couldn't be found |
cd8116e | Do not extract files containing a colon in the name |
Is there anything I need/can do? What is the procedure here (this is my first contribution to sage...)?
Is this for WSL 1 or WSL 2? Also, my understanding was that symlinks work better now in WSL 2 but I haven't tried it or looked into it in detail.
It's for WSL 1. As far as I know, WSL 2 is still in preview (or at least requires a insiders build of Windows).
The problem with the symbolic links could have been caused by the fact that I cloned the repository under Windows, which replaced the symbolic links with text files. It might work if you clone under WSL.
Replying to @tobiasdiez:
The problem with the symbolic links could have been caused by the fact that I cloned the repository under Windows, which replaced the symbolic links with text files. It might work if you clone under WSL.
Indeed, I think any build process for Sage on WSL should be 100% under WSL. This is likely also the cause of some of the permission problems you noted.
Ticket retargeted after milestone closed
Using git in Windows with git config --global core.symlinks true may help. See #29295
I cannot view the branch on trac.
https://github.com/sagemath/sagetrac-mirror/commits/public/wls
and it looks quite outdated.
Rebase?
It might outdated because the original commits are now already 7 months old. I would appreciate if I could get advice concerning the open points raised above. I don't want to rebase and then have to wait another year to get feedback. (Sorry for the slightly aggressive tone but honestly it wasn't a great experience so far contributing to sage.)
I think guidance regarding symlinks and lineendings when checking out on Windows should be added to the Installation Manual / Developer Manual.
We now have #29513 sage-spkg: Remove check for executable permission on spkg-install.in script templates
The commit "Do not extract files containing a colon in the name" needs a better solution. Upstream bug report to avoid non-portable file names
Replying to @tobiasdiez:
It might outdated because the original commits are now already 7 months old. I would appreciate if I could get advice concerning the open points raised above.
Trac notifications by email are not very reliable, unfortunately, so you may have missed comments 6 and 8 above.
The zlib problem is likely not specific to WSL at all. Since 9.0 we have implemented lots of portability fixes for various distributions, including Ubuntu.
Replying to @tobiasdiez:
(Sorry for the slightly aggressive tone but honestly it wasn't a great experience so far contributing to sage.)
apologies, this reflects the number of active contibutors to "core" Sage who use WSL (or Windows in general), unfortunately.
At least now we have started to test on Windows using GitHub Actions, so if we managed this with WSL too we would be in a good shape here.
So here's what we need:
- #29521 - to fix the colons
- guidance regarding symlinks and lineendings when checking out on Windows should be added to the Installation Manual / Developer Manual (where git operation is discussed)
- perhaps a check in configure to reject source trees that have been checked out with a wrong symlinks/lineendings mode. This would also be useful for Cygwin.
I don't think automatic testing on WSL is necessary. WSL1 is really just a stock Linux distribution running on a slightly weird filesystem.
regarding the need to test on WSL, there was a report on an error (off by 1 in some combinatorics) on WSL of a test with method 'bober' in it.
here:
https://groups.google.com/forum/m/#!search/bober$20wsl/sage-devel/zuPzRgpSw3s
sage: number_of_partitions(241, algorithm='bober')
114540884553039
sage: number_of_partitions(241) # this one is correct
114540884553038
I'm astonished about all those complications reported in this ticket. After I had installed the Ubuntu 18.04 app under WSL2, I followed the Sage installation instructions for Ubuntu (using the recently communicated prerequisites). As far as I remember (it's just some weeks ago) there was no larger amount of troubleshooting to do than I usually do under Linux Mint (some LaTeX sty-files missing ...). But I did that below my home directory inside Ubuntu.
What has been the reason to clone the sources into the Windows filesystem? My feeling is that most of the trouble reported here comes from that fact.
Concerning the installation of Sage binaries the instructions for Ubuntu work without any trouble, too (I did it under /opt/). The most easiest way to have a running Sage under WSL is via sudo apt-get sagemath. But this is an older version (8.1). The bug I have reported on
sage-devel (see comment 22 above) is reproducible under the stable 9.0 binaries, as well, but not under the apt version.
Generally speaking, I am really impressed about the high compatibility with proper Ubuntu. The most annoying thing for me is that you can't use FUSE based mount options like sshfs, encfs .. (and it seems that they don't have a plan to fix that).
But I think, we should not announce to use Sage under WSL before all doctests are fixed. Here is the list from the log file attached to the sage-devel thread (the first item being the issue from comment 22):
sage -t --warn-long 44.8 src/sage/combinat/partitions.pyx # 8 doctests failed
sage -t --warn-long 44.8 src/sage/docs/conf.py # 1 doctest failed
sage -t --warn-long 44.8 src/sage/doctest/test.py # 2 doctests failed
sage -t --warn-long 44.8 src/sage/libs/glpk/error.pyx # 1 doctest failed
sage -t --warn-long 44.8 src/sage/matrix/matrix_mod2_dense.pyx # Killed due to segmentation fault
sage -t --warn-long 44.8 src/sage/misc/sagedoc.py # 4 doctests failed
sage -t --warn-long 44.8 src/sage/numerical/backends/glpk_backend.pyx # 1 doctest failed
sage -t --warn-long 44.8 src/sage/rings/number_field/number_field_ideal_rel.py # 1 doctest failed
if glpk comes from Ubuntu, then these libs/glpk/error.pyx and numerical/backends/glpk_backend.pyx are expected failures, due to Sage's patches.
Thanks for the feedback.
I think checking out the code under Windows should be supported. I want to develop under Windows using my normal tool set, and only run the compilation under Linux. Although the file system support of WSL got better over time, transferring or accessing files that are in the Linux system from Windows is still an hassle.
Replying to @mkoeppe:
Using git in Windows with
git config --global core.symlinks truemay help. See #29295
That might work.
Replying to @tobiasdiez:
Thanks for the feedback.
I think checking out the code under Windows should be supported. I want to develop under Windows using my normal tool set, and only run the compilation under Linux.
Needless to say, running the result of the compilation, debugging, etc. will also happen on WSL.
It seems that at best we talk about editing the sources and using git on Windows
in such a setup.
Replying to @tobiasdiez:
Thanks for the feedback.
I think checking out the code under Windows should be supported. I want to develop under Windows using my normal tool set, and only run the compilation under Linux.
Yes, this would be nice and will need some work.
Although the file system support of WSL got better over time, transferring or accessing files that are in the Linux system from Windows is still an hassle.
Replying to @mkoeppe:
Using git in Windows with
git config --global core.symlinks truemay help. See #29295That might work.
Well, as a first step, would you like to help document what already works for our upcoming 9.1 release?
Changing the Sage build so it works with non-Unix lineendings and restructuring the build so that no symlinks are in the source repository [obviously, not by making copies...] is more work, which won't be ready for the 9.1 release, but would be a valuable contribution.
Replying to @dimpase:
if glpk comes from Ubuntu, then these
libs/glpk/error.pyxandnumerical/backends/glpk_backend.pyxare expected failures, due to Sage's patches.
But at least those tests which lead to mathematical wrong answers should be fixed or marked with a note that they are not validated under WSL.
Replying to @tobiasdiez:
Thanks for the feedback.
I think checking out the code under Windows should be supported. I want to develop under Windows using my normal tool set, and only run the compilation under Linux. Although the file system support of WSL got better over time, transferring or accessing files that are in the Linux system from Windows is still an hassle.
Did you try to synchronize directories between WSL and Windows (for example using Unison)?
Replying to @dimpase:
Replying to @tobiasdiez:
Thanks for the feedback.
I think checking out the code under Windows should be supported. I want to develop under Windows using my normal tool set, and only run the compilation under Linux.
Needless to say, running the result of the compilation, debugging, etc. will also happen on WSL.
It seems that at best we talk about editing the sources and using git on Windows
in such a setup.
Exactly!
Replying to @soehms:
But at least those tests which lead to mathematical wrong answers should be fixed or marked with a note that they are not validated under WSL.
Definitely. The failure with number_of_partitions(241, algorithm='bober') should become a separate ticket. Is this already a doctest?
Replying to @mkoeppe:
Is this already a doctest?
Yes, see the first line of the list shown in comment 24. I've just simplified that to it's first occurence which is reproducible 241 on both 9.1.rc0 and on stable 9.0 installed from binaries.
But, I have to correct my statement from comment 24, that the bug would not occur on version 8.1. That is not true, but here, 241 it not in the list of failures (see ticket #29600 I've created for that issue).
Friedrich, I hope you had a good start at Bosch this month and time enough left to contribute your experience on doing Sage under WSL.
Thanks, Sebastian!
I think I also used WSL2 when building Sage in it. At least my experience was as yours: not more problems than under a normal Ubuntu. (However I also did everything within the WSL and tried to leave out Windows as much as possible.) So not sure how much I can help here..
Just for reference: my google-groups post re my WSL try is here https://groups.google.com/forum/#!searchin/sage-devel/wsl|sort:date/sage-devel/WoEWxYmwQnI/aeqy6YxoDQAJ
Let's close this ticket in favor of #25206 (Update README, installation manual and developer's guide on Windows (WSL and Cygwin))
Changed keywords from windows, wls, make to windows, wsl, make
Changed author from Tobias Diez to none
Reviewer: Tobias Diez
I'm -1 on removing symlinks from the repository just to support git checkout on Windows (which does support symlinks in WSL). If you want to develop Sage on Windows I think you should also use git through WSL and not using the native Windows git. Same goes when using Cygwin (where this works fine).
diff --git a/build/sage_bootstrap/package.py b/build/sage_bootstrap/package.py
index aff9212..3ee6c6a 100644
--- a/build/sage_bootstrap/package.py
+++ b/build/sage_bootstrap/package.py
@@ -229,9 +229,11 @@ class Package(object):
self.__md5 = result.get('md5', None)
self.__sha1 = result.get('sha1', None)
self.__cksum = result.get('cksum', None)
- self.__tarball_pattern = result['tarball']
+ self.__tarball_pattern = result.get('tarball', "")
# Name of the directory containing the checksums.ini file
self.__tarball_package_name = os.path.realpath(checksums_ini).split(os.sep)[-2]
+ if not result.get('tarball'):
+ log.warn("Could not find checksum in %s" % checksums_ini)This can be written just result.get('tarball'), without specifying a default. In this case the default is already None. Then below you can write if not self.__tarball_pattern:.
Out of curiosity though why would this happen? Shouldn't this always be here? And why an explicit check for 'tarball' and not for the other required fields?
diff --git a/build/sage_bootstrap/uncompress/tar_file.py b/build/sage_bootstrap/uncompress/tar_file.py
index cf707e7..b6d8dc9 100644
--- a/build/sage_bootstrap/uncompress/tar_file.py
+++ b/build/sage_bootstrap/uncompress/tar_file.py
@@ -116,6 +116,9 @@ class SageBaseTarFile(tarfile.TarFile):
The additional ``**kwargs`` are for Python 2/3 compatibility, since
different versions of this method accept additional arguments.
"""
+ if ":" in targetpath:
+ return
+
old_umask = os.umask(self.umask)
try:
super(SageBaseTarFile, self)._extract_member(tarinfo, targetpath,This check is overly broad since this will work on any POSIX system (and should work on WSL too but I understand it's presently a limitation with the NTFS filesystem).
Instead perhaps it should try extracting the file anyways but properly handle whatever exception occurs when trying to extract it on WSL, and produce a warning as well.
This is a laudable effort and I want to support it, but I think as-is it's making too many changes to support this one one build environment, to the detriment of all other build environments (especially the removal of symlinks). I think some of the git-related problems also stem from mixing native Windows and WSL gits, but please correct me if I'm wrong on that. I haven't tried WSL since WSL2 came out some I'm not completely versed in what limitations exist or not (but I'm quite sure I recall it supporting symlinks just fine).
I don't have access to a fast Windows machine right now, but when I get back to my office I will, and then I can try to help with this.
Note the milestone on this ticket...
D'oh. Sorry for the noise. Regardless, it looks like there might have been some useful things in here? Or is the colon-in-filename issue resolved now?
Yes, it's all done, see for example comment 21.
Thanks embray for the review. This issue is indeed somewhat old, I've described how to set up a working build envirnmont using WSL here: #25206 comment:63