sage-download-file: Fix certificate problems with https downloads from upstream_url when sage-system-python is XCode's python3 on macOS
mkoeppe opened this issue · 42 comments
#26351 added an optional upstream_url field to build/pkgs/*/checksums.ini. It streamlines the procedure for testing upgrade tickets: Developers or automatic testing facilities can pass an extra flag -o to sage-spkg to allow downloading from upstream rather than from Sage mirrors (where the updated ticket will be made available later only).
Many upstream package URLs use the https protocol - in contrast to the http protocol used when downloading from the Sage mirrors. The downloading is done via build/bin/sage-download-file, which uses the urllib module. It supports the https protocol.
However, SSL certificate problems are common on test systems. For example, if one uses XCode's python3 as the system python, then urllib does not automatically uses the standard system certificates. (This is apparently a known issue -- which is considered "wontfix" by Apple as reported here:
HandBrake/HandBrake#2216 (comment))
We add an option --no-check-certificate to sage-download-file, disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification).
Developers can set this option using the environment variable SAGE_DOWNLOAD_FILE_OPTIONS when installing packages (either by make or by using sage -i).
We note that even with SSL certificates disabled, there is still cryptographic protection because of the checksums recorded in checksums.ini.
Other possible workarounds considered:
- switching from using urllib directly to the requests library
- passing cafile=..., capath=... to urllib, perhaps coming from an environment variable
- using python instead of python3 on macOS as system python
As of #29090 (sage-system-python fixup) prefers /usr/bin/python3 over /usr/bin/python, leading to:
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
(see https://github.com/mkoeppe/sage/runs/538432620)
CC: @vbraun @dimpase @kiwifb @jhpalmieri @videlec @fchapoton @kliem
Component: build
Author: Matthias Koeppe
Branch: 90ea00b
Reviewer: Jonathan Kliem
Issue created by migration from https://trac.sagemath.org/ticket/29418
Description changed:
---
+++
@@ -2,3 +2,10 @@
As seen with `local-homebrew-macos-minimal` in https://github.com/mkoeppe/sage/runs/538432620
+As of #29090 (sage-system-python fixup) prefers `/usr/bin/python3` over `/usr/bin/python`, leading to:
+
+```
+ File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
+ self._sslobj.do_handshake()
+OSError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
+```Description changed:
---
+++
@@ -1,6 +1,13 @@
This is for downloading from an https `upstream_url` when the host is poorly configured.
-As seen with `local-homebrew-macos-minimal` in https://github.com/mkoeppe/sage/runs/538432620
+In particular this happens with XCode's `python3` as the system python. This is apparently a known issue -- which is considered "wontfix" by Apple as reported here:
+https://github.com/HandBrake/HandBrake/issues/2216#issuecomment-527114519
+
+We should work around this by either:
+- switching from using urllib directly to the requests library
+- passing cafile=..., capath=... to urllib, perhaps coming from an environment variable
+- disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification), perhaps triggered by an option --no-check-certificate to sage-download-file
+- using python instead of python3 on macOS as system python
As of #29090 (sage-system-python fixup) prefers `/usr/bin/python3` over `/usr/bin/python`, leading to:
@@ -9,3 +16,6 @@
self._sslobj.do_handshake()
OSError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
```
+(see https://github.com/mkoeppe/sage/runs/538432620)
+
+that's why I suggested not to bother with MacOS/Xcode's python(3).
Can't one get Python3 from python.org in MacOS by running a script?
New commits:
bd24641 | build/bin/sage-download-file: Add option --no-check-certificate |
Description changed:
---
+++
@@ -1,12 +1,15 @@
-This is for downloading from an https `upstream_url` when the host is poorly configured.
+We add an option `--no-check-certificate` to `sage-download-file`, disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification).
+
+Users can set this option using the environment variable SAGE_DOWNLOAD_FILE_OPTIONS when using `sage -i`. This is for enabling them to download from an https `upstream_url` when the host is poorly configured.
In particular this happens with XCode's `python3` as the system python. This is apparently a known issue -- which is considered "wontfix" by Apple as reported here:
https://github.com/HandBrake/HandBrake/issues/2216#issuecomment-527114519
-We should work around this by either:
+---
+
+Other possible workarounds considered:
- switching from using urllib directly to the requests library
- passing cafile=..., capath=... to urllib, perhaps coming from an environment variable
-- disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification), perhaps triggered by an option --no-check-certificate to sage-download-file
- using python instead of python3 on macOS as system python
As of #29090 (sage-system-python fixup) prefers `/usr/bin/python3` over `/usr/bin/python`, leading to:Changed branch from u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3 to none
Author: Matthias Koeppe
Replying to @dimpase:
that's why I suggested not to bother with MacOS/Xcode's python(3).
Can't one get Python3 from python.org in MacOS by running a script?
Please... the idea is to make Sage installation easier, not harder.
One often runs into these certificate errors also with random Linux images when I run them on Docker. So this workaround is valuable for the automatic testing in any case.
Testing this at https://github.com/mkoeppe/sage/actions/runs/65346154
New commits:
bd24641 | build/bin/sage-download-file: Add option --no-check-certificate |
e143939 | build/bin/sage-spkg: Append to SAGE_DOWNLOAD_FILE_OPTIONS instead of overwriting it |
Replying to @mkoeppe:
Replying to @dimpase:
that's why I suggested not to bother with MacOS/Xcode's python(3).
Can't one get Python3 from python.org in MacOS by running a script?Please... the idea is to make Sage installation easier, not harder.
well, it seems that using system's python would make a slighly cryptographically
broken Sagemath, one way or another.
And, of course, the present ticket is about sage-system-python, which is only used for bootstrapping and build. Unrelated to the use of a system python3 for venv (#27824), which has a completely separate test for what it accepts.
oy gevalt, pythons everywhere!
Replying to @dimpase:
oy gevalt, pythons everywhere!
This will be addressed in #45678 "switch Sage from Python to Julia"
Rebased, needs review
Okay, I'm really confused, or my Sage installation is broken, or both. If I use ./sage -i beautifulsoup4 with a build using the system Python on OS X, it fails, but not because of the issues here. Instead, I see this:
make[2]: *** No rule to make target `/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/SYSTEM/sage-9.1.beta9/local/var/lib/sage/installed/beautifulsoup4-none', needed by `all-sage'. Stop.
If instead I run make beautifulsoup4, it succeeds. When I ran ./sage -i tornado it didn't seem to even try to install it, but ./sage -f tornado worked.
So: why does ./sage -i ... not work? And what is the point of this ticket: what problem does it solve?
Could you try if the problem goes away if you run bootstrap?
Description changed:
---
+++
@@ -1,6 +1,6 @@
We add an option `--no-check-certificate` to `sage-download-file`, disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification).
-Users can set this option using the environment variable SAGE_DOWNLOAD_FILE_OPTIONS when using `sage -i`. This is for enabling them to download from an https `upstream_url` when the host is poorly configured.
+Users can set this option using the environment variable SAGE_DOWNLOAD_FILE_OPTIONS when installing packages (either by `make` or by using `sage -i`). This is for enabling them to download from an https `upstream_url` when the host is poorly configured.
In particular this happens with XCode's `python3` as the system python. This is apparently a known issue -- which is considered "wontfix" by Apple as reported here:
https://github.com/HandBrake/HandBrake/issues/2216#issuecomment-527114519If I run bootstrap, then running ./sage -i benzene acts like ./sage -i tornado: it doesn't install. There is a message
running ./configure '--enable-beautifulsoup4' '--enable-biopython' '--enable-benzene'
and then if I search my terminal window for benzene, the only other hit is
benzene-20130630: does not support check for system package; optional, use "./configure --enable-benzene" to install
I don't think I tested #29113 well enough when I was reviewing it.
This fixes it for me:
diff --git a/src/bin/sage b/src/bin/sage
index 10acddcd96..6ccff3789c 100755
--- a/src/bin/sage
+++ b/src/bin/sage
@@ -404,7 +404,7 @@ if [ "$1" = '-i' ]; then
# 'CC=gcc -Wall' '--enable-e_antic'
CONFIG_CMD="./configure $(./config.status --config) $ENABLE_ARGS"
echo >&2 "running $CONFIG_CMD"
- bash -c "$CONFIG_CMD" && $MAKE all-build
+ bash -c "$CONFIG_CMD" && $MAKE $PACKAGES
else
echo "New packages may have been installed."
echo "Re-running configure and make in case any dependent packages need updating."(or maybe it could be $MAKE all-build $PACKAGES?)
By the way, I still don't understand the goal of this ticket.
Let's please take the issues with sage -i to the new ticket #29481. Thanks for catching this.
Description changed:
---
+++
@@ -1,9 +1,15 @@
+#26351 added an optional `upstream_url` field to `build/pkgs/*/checksums.ini`. It streamlines the procedure for testing upgrade tickets: Developers or automatic testing facilities can pass an extra flag `-o` to `sage-spkg` to allow downloading from upstream rather than from Sage mirrors (where the updated ticket will be made available later only).
+
+Many upstream package URLs use the `https` protocol - in contrast to the `http` protocol used when downloading from the Sage mirrors. The downloading is done via `build/bin/sage-download-file`, which uses the `urllib` module. It supports the https protocol.
+
+However, SSL certificate problems are common on test systems. For example, if one uses XCode's `python3` as the system python, then `urllib` does not automatically uses the standard system certificates. (This is apparently a known issue -- which is considered "wontfix" by Apple as reported here:
+https://github.com/HandBrake/HandBrake/issues/2216#issuecomment-527114519)
+
We add an option `--no-check-certificate` to `sage-download-file`, disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification).
-Users can set this option using the environment variable SAGE_DOWNLOAD_FILE_OPTIONS when installing packages (either by `make` or by using `sage -i`). This is for enabling them to download from an https `upstream_url` when the host is poorly configured.
+Developers can set this option using the environment variable SAGE_DOWNLOAD_FILE_OPTIONS when installing packages (either by `make` or by using `sage -i`).
-In particular this happens with XCode's `python3` as the system python. This is apparently a known issue -- which is considered "wontfix" by Apple as reported here:
-https://github.com/HandBrake/HandBrake/issues/2216#issuecomment-527114519
+We note that even with SSL certificates disabled, there is still cryptographic protection because of the checksums recorded in `checksums.ini`.
---
Replying to @jhpalmieri:
By the way, I still don't understand the goal of this ticket.
Reworked the ticket description, please take a look
Replying to @mkoeppe:
Replying to @jhpalmieri:
By the way, I still don't understand the goal of this ticket.
Reworked the ticket description, please take a look
Sorry, I wasn't clear enough. What system configuration and commands lead to the problem in the description? I can't reproduce the problem, so I can't tell if the solution here works.
From what I understand this is a problem that only occurs when one has python3 from Xcode. This doesn't ship certify and never will (at least Apple doesn't have any intentions). So it is really difficult to establish an ssl connection with that.
Do you have successfully tested this ticket?
Do I understand correctly that the problem occurring could be theoretically fixed by manually downloading the correct packages into the upstream folder? (Yes this is not a good approach for testing environments.)
Also this flag would never be needed for a normal user as long as the sage mirrors don't use ssl?
Replying to @kliem:
From what I understand this is a problem that only occurs when one has python3 from Xcode. This doesn't ship certify and never will (at least Apple doesn't have any intentions). So it is really difficult to establish an ssl connection with that.
That's a correct description for macOS. But the problem also appears on Linux distributions if one does not install ca-certificates, or those are outdated.
Do you have successfully tested this ticket?
Yes, I have been using this as part of #29417 since end of March.
Do I understand correctly that the problem occurring could be theoretically fixed by manually downloading the correct packages into the upstream folder? (Yes this is not a good approach for testing environments.)
That's correct for the macOS tests using tox local. For the Linux tests with tox docker note that /upstream is in .gitignore (therefore in .dockerignore) and is therefore not provided to the container.
Also this flag would never be needed for a normal user as long as the sage mirrors don't use ssl?
That's right, normal users would not use the upstream_url at all; it is a feature for developers only.
Reviewer: Jonathan Kliem
LGTM.
Btw, I have been using this ticket to test my tickets, e.g. here: https://github.com/kliem/sage-test-27122/actions/runs/72102779
Thank you!