Deprecation of non-SNI compatible clients.
Closed this issue · 61 comments
Our CDN provider is deprecating non-SNI support for access to pypi.org and files.pythonhosted.org. It appears that this rollout has begun and users without SNI support can expect disruptions.
Brownout windows
| Dates | Window(s) |
|---|---|
| March 31st - April 7th | :00-:10 every hour |
| April 7th - April 14th | :00-:15 every hour |
| April 14th - April 21st | :00-:15 and :30-:40 every hour |
| April 21st - April 28th | :00-:15 and :30-:45 every hour |
| April 28th - May 3rd | :00-20 and :30-:50 every hour |
| May 3rd onward | Support for non-SNI clients disabled in our VCL |
| May 5th | Fastly will migrate our host, disabling non-SNI clients completely |
Note transitions between phases will occur at 12:00:00 UTC. All but the last phase are fully automated.
Until the migration is complete, impacted users can anticipate seeing this error when the brownout is in effect:
That was generated using Python 2.6 and pip 9.0.3, the output to terminal may vary, but the error message is:
[[[!!! BREAKING CHANGE !!!]]] Support for clients that do not support Server Name Indication is temporarily disabled and will be permanently deprecated soon. See https://status.python.org/incidents/hzmjhqsdjqgb and https://github.com/pypa/pypi-support/issues/978 [[[!!! END BREAKING CHANGE !!!]]]
Update (2021-03-31)
In order to provide visibility for this deprecation and give people a chance to test their fixes we will be implementing rolling brownouts of non-SNI support beginning later today. The mechanism will follow similar pattern to our TLS1.0/1.1 deprecation.
See above for status of these brownouts
Update (2021-03-26)
After the initial rollout began, a number of users reached out to Fastly directly, which led to them temporarily rolling back the change. During the rollback a certificate issue was encountered that required re-verifying our domain. This lead to a number of similar but only slightly related issues, as well as quite a bit of confusion for everyone involved.
We now have a new date for rollout of deprecation for non-SNI compatible clients.
Between May 3rd and May 6th 2021, a more precise date will be available as the window nears.
At some point during that week, non-SNI clients will stop being able to access pypi.org and files.pythonhosted.org.
(end of Update 2021-03-26)
This impacts approximately:
- .37% of traffic for *.pythonhosted.org
- .000016% of traffic for pythonhosted.org
- .000005 of traffic for *pypi.org
Due to the small portion of traffic that is impacted, the PyPI administrators and PSF Infrastructure will not be seeking a work around or alternative CDN.
Primarily impacted are pip users running Python versions less than 2.7.9.
For users of Python 2.7.{0...8}
Upgrading to the last Python 2.7 release is an option.
However, note that Python 2.7 series itself is now End of Life and support in pip was dropped with version 21.0.
For users of Python 2.6.x and lower:
Neither the Python core developers, or pip maintainers support Python 2.6 and below.
If someone is aware of a work around for this issue (SNI support specifically) they are welcome to share it here for others.
There is no recommended solution from the PyPI team.
I like to point out that RHEL 7 and CentOS 7 users are not affected. Python 2.7.5 on RHEL/CentOS 7 contains a backport of the ssl module improvement from Python 2.7.9.
Users of OpenSSL < 0.9.8f don't have SNI support either. Your Python must be built with a more recent version of OpenSSL.
You can check if your Python and OpenSSL version supports SNI with python2 -c "import ssl; print(ssl.HAS_SNI)".
Related: pypa/pip#4350
SNI support was apparently re-enabled today.
@felixfontein Yes, see my comment here: #974 (comment)
FYI CentOS 7 ships with python 2.7.5, I think this will break many people's setups.
FYI CentOS 7 ships with python 2.7.5, I think this will break many people's setups.
CentOS 7 is fine. It has Python 2.7.5 + a bunch of patches that enables SNI support. See #978 (comment)
I like to point out that RHEL 7 and CentOS 7 users are not affected. Python 2.7.5 on RHEL/CentOS 7 contains a backport of the ssl module improvement from Python 2.7.9.
Users of OpenSSL < 0.9.8f don't have SNI support either. Your Python must be built with a more recent version of OpenSSL.
You can check if your Python and OpenSSL version supports SNI with
python2 -c "import ssl; print(ssl.HAS_SNI)".
I got this earlier today:
[root@5f1e14832ee8 /]# cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)
[root@5f1e14832ee8 /]# python2 --version
Python 2.7.5
[root@5f1e14832ee8 /]# python2 -c "import ssl; print(ssl.HAS_SNI)"
True
but then:
distutils.errors.DistutilsError: https://files.pythonhosted.org/packages/65/e2/8cb5e718a3a63e8c22677fde5e3d8d18d12a551a1bbd4557217e38a97ad0/pbr-5.5.1.tar.gz#sha256=5fad80b613c402d5b7df7bd84812548b2a61e9977387a80a5fc5c396492b13c9 hostname 'files.pythonhosted.org' doesn't match either of 'r.ssl.fastly.net', '*.catchpoint.com', '*.cnn.io', '*.dollarshaveclub.com', '*.eater.com', '*.fastly.picmonkey.com', '*.files.saymedia-content.com', '*.ft.com', '*.meetupstatic.com', '*.nfl.com', '*.pagar.me', '*.picmonkey.com', '*.realself.com', '*.sbnation.com', '*.shakr.com', '*.streamable.com', '*.surfly.com', '*.theverge.com', '*.thrillist.com', '*.vox-cdn.com', '*.vox.com', '*.voxmedia.com', 'eater.com', 'ft.com', 'i.gse.io', 'picmonkey.com', 'realself.com', 'static.wixstatic.com', 'streamable.com', 'surfly.com', 'theverge.com', 'vox-cdn.com', 'vox.com', 'www.joyent.com'
from a setuptools invocation. But it's working now, so what's happening?
@dpanech-wr fastly temporarily re-enabled SNI for files.pythonhosted.org, see #974 (comment).
I had success with appending --trusted-host pypi.org --trusted-host files.pythonhosted.org to my pip install commands, though I have not looked at whether there's security issues around that
Hello I encountered this error with python-2.7.13 which says it supports SNI:
# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
# python --version
Python 2.7.13
# python -c "import ssl; print(ssl.HAS_SNI)"
True
Yet I was getting this error:
19:57:41 [INFO] distutils.errors.DistutilsError: https://files.pythonhosted.org/packages/d9/c5/396c2c06cc89d4ce2d8ccf1d7e6cf31b33d4466a7c65a67a992adb3c6f29/setuptools-git-1.2.tar.gz#sha256=ff64136da01aabba76ae88b050e7197918d8b2139ccbf6144e14d472b9c40445 hostname 'files.pythonhosted.org' doesn't match either of 'r.ssl.fastly.net', '*.catchpoint.com', '*.cnn.io', '*.dollarshaveclub.com', '*.eater.com', '*.fastly.picmonkey.com', '*.files.saymedia-content.com', '*.ft.com', '*.meetupstatic.com', '*.nfl.com', '*.pagar.me', '*.picmonkey.com', '*.realself.com', '*.sbnation.com', '*.shakr.com', '*.streamable.com', '*.surfly.com', '*.theverge.com', '*.thrillist.com', '*.vox-cdn.com', '*.vox.com', '*.voxmedia.com', 'eater.com', 'ft.com', 'i.gse.io', 'picmonkey.com', 'realself.com', 'static.wixstatic.com', 'streamable.com', 'surfly.com', 'theverge.com', 'vox-cdn.com', 'vox.com', 'www.joyent.com'
As per #974 (comment) this issue is no longer present after Fastly "fixed it"
@dpanech-wr and @andrewegel
could you please post your pip version and how you installed pip? Are you using pip from your OS vendor or an updated pip? Debian and RHEL contain modified versions of pip with de-vendored dependencies. They don't behave like upstream versions of pip.
Recent pip works for me on RHEL 7.9. tcpdump confirms that Python and pip are sending a SNI TLS extension in ClientHello. The first request has Server Name: pypi.org. The second request has Server Name: files.pythonhosted.org
# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
...
# python2 --version
Python 2.7.5
# curl -O https://bootstrap.pypa.io/pip/2.7/get-pip.py
# python2 get-pip.py
...
# pip2 --version
pip 20.3.4 from /usr/lib/python2.7/site-packages/pip (python 2.7)
# python2 -c "import ssl; print(ssl.HAS_SNI, ssl.OPENSSL_VERSION)"
(True, 'OpenSSL 1.0.2k-fips 26 Jan 2017')
# pip2 install ...
could you please post your pip version and how you installed pip? Are you using pip from your OS vendor or an updated pip?
Latest was from curl -fSL "https://bootstrap.pypa.io/pip/2.7/get-pip.py" | python:
# which pip
/usr/local/bin/pip
# pip --version
pip 20.3.4 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)
But this also had an issue with the OS vendors, which was:
# dpkg -l | grep pip
ii python-pip 9.0.1-2+deb9u2 all Python package installer
ii python-pip-whl 9.0.1-2+deb9u2 all Python package installer
# pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
# python --version
Python 2.7.13
# python -c "import ssl; print(ssl.HAS_SNI)"
True
BOTH had issues, irrespective of using the OS vendors or one that was installed from https://bootstrap.pypa.io/pip/2.7/get-pip.py
Recent pip works for me on RHEL 7.9. tcpdump confirms that Python and pip are sending a SNI TLS extension in ClientHello. The first request has Server Name: pypi.org. The second request has Server Name: files.pythonhosted.org
Yeah, but its been fixed now as per this comment: #974 (comment)
This is me logging an issue that before the fix in #974 (comment) was completed, I ran into this issue.
For me:
[root@9c11c1209e95 ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
[root@9c11c1209e95 ~]# python --version
Python 2.7.5
[root@9c11c1209e95 ~]# rpm -q python
python-2.7.5-88.el7.x86_64
[root@9c11c1209e95 ~]# pip --version
pip 8.1.2 from /usr/lib/python2.7/site-packages (python 2.7)
[root@9c11c1209e95 ~]# rpm -q python2-pip
python2-pip-8.1.2-12.el7.noarch
These are not the latest packages from the vendor.
In any case, both the vendor pip and https://bootstrap.pypa.io/pip/2.7/get-pip.py work right now.
But I did get a "hostname 'files.pythonhosted.org' doesn't match either of" error yesterday.
But I did get a "hostname 'files.pythonhosted.org' doesn't match either of" error yesterday.
That's peculiar. According to Wireshark, Python 2.7.5 and pip 8.1.2 from EPEL are sending a correct SNI extension for files.pythonhosted.org.
I just received an email from Amazon Support that said that they had worked this out with Fastly. They declined to provide any details. @ewdurbin, can you confirm?
What about python3 clients? I have Python 3.7.6 Anaconda and today I've found problems with pip. pip version -- pip 20.0.2
Please see the update (2021-03-26) in the top comment.
@mdw123 I don't have any detials about any conversations Amazon Support and Fastly.
@potassium-chloride without further details on what issues you are having I can't help determine if it is in any way related to this problem. Please file a separate issue.
This impacts approximately:
.37% of traffic for *.pythonhosted.org
.000016% of traffic for pythonhosted.org
.000005 of traffic for *pypi.org
Due to the small portion of traffic that is impacted, the PyPI administrators and PSF Infrastructure will not be seeking a work around or alternative CDN.Primarily impacted are pip users running Python versions less than 2.7.9.
The thing you don't consider in this analysis is the impact. For me, this took down my staging system because of a component AWS includes that I have little control over. Had I deployed prod, it would have taken prod out for several hours. I manually upgrade pip & then setup tools when I create a new instance, but I don't have a good way to validate if I can bootstrap to a new version with pip once fastly deprecates non-SNI requests. The fact that "a number of users reached out to Fastly directly" (I was one) speaks to the significance of this incident for those people, as well as their ability to understand exactly what changed.
I'm still pushing AWS on this, but I don't think the small numbers should give you much comfort regarding the impact.
Root cause is that Fastly wants more money for non-SNI setups & python.org isn't willing to pay? How much do they want?
(these are my personal views, and not like an official statement from someone or whatever)
Root cause is that Fastly wants more money for non-SNI setups and python.org isn't willing to pay?
No.
Fastly is an infrastructure sponsor of the PSF, which means that they donate infrastructure to the PSF. You can see they're listed on https://pypi.org/sponsors.
The Python Software Foundation is not paying Fastly for the caching they provide for various PSF properties, like python.org and pypi.org.
Because you brought it up, the root cause is that you're using versions of Python/pip that are no longer supported by their upstream maintainers, and not investing in keeping them functional - either by getting that Python from an enterprise distribution (i.e. paying for support) or by investing in patching to modern versions of OpenSSL on those Python versions (i.e. paying software engineers to do it).
What's happening here is Fastly is dropping support for older, less secure protocols. And however you're getting Python, it has not been patched to have newer more secure protocols that have been around for many years now.
I do understand and empathise that you're concerned about your deployment pipeline. But, if your pipeline depends on other things on the Internet staying online and those things supporting all kinds of legacy protocols for versions of OSS stuff that isn't even supported upstream... I'd suggest:
- adjusting your expectations around how long unsupported open source software will work.
- invest in insulating your deployment/build pipelines from external resources changing.
- upgrade to newer Python versions or get enterprise support for using your older versions of Python.
I can't say that anything @pradyunsg said is wrong.
Of course we recognize that analysis based on access patterns alone will not elucidate every niche use case. However in general, it appears that people misunderstand the resources available to handle each of those niches, even if we had a perfect understanding of them.
PyPI and the PSF Infrastructure have around 10 total contributors on an ongoing basis, 3 active administrators, and 1-part-of-1 full-time staff member dedicated to them. Because of this, decisions on committing resources and time to supporting out of date and end of life protocols and clients can be difficult. In this instance it was made easier by the fact that the most recent and widely deployed non-SNI series Python (2.6) left support in 2013-10-29.
As a general rule for protocols that relate to the security and health of the internet at large, we will continue to adopt new standards as soon as possible and deprecate old standards in line with our providers. As they are profit seeking companies, we trust their judgment on when it is time to sunset out of date protocols.
Thanks for the response & clarification of the relationship between Fastly and PyPI @ewdurbin & @pradyunsg. When I contacted Fastly when things started breaking, they suggested I speak with my account manager, which is the beginning of a commercial, rather than a technical conversation. I am grateful for the work that goes into the platform. I have upgraded my systems using the tools at my disposal, I am just unclear as to whether that will work once non-SNI requests are dropped for good in May. At the same time, AWS is asserting that everything is fine/will be fine with no description of exactly what their plan is. Clearly, the action to solve this sits with me.
I looked in to the situation a bit based on @andrewegel's earlier post concerning Debian 9. As he indicated, all the python versions in question do seem to support SNI. However, it turns out that the setuptools package itself is the important one here. Setuptools added SNI support with version 36.8.0 in pull request #1190
Distributions that are running setuptools < 36.8.0 that have not patched it for SNI support will break when Fastly disables support for non-SNI clients, regardless of what python version they use. This includes Debian 9 and Amazon Linux 1. I am following up to get fixes in place in both of those distributions prior to Fastly's cutoff date.
I have not investigated what versions of RHEL, CentOS, Ubuntu, or other distributions may be impacted and will leave that to the community.
Distributions that are running setuptools < 36.8.0 that have not patched it for SNI support will break when Fastly disables support for non-SNI clients, regardless of what python version they use.
Note that this is only an issue, if they're also depending on some setuptools-specific functionality (like setup_requires) or anything that uses setuptools as a package manager. That's basically invoking easy_install's code paths internally.
Those features have basically been deprecated/strong discouraged for multiple years now, and I imagine that any fixes for these would need to happen from the vendor/distro containing the really-old-version of setuptools.
I checked Python 2.7.5 on RHEL 7.9 with tcpdump. pip requests to pypi and pythonhosted are sending the SNI TLS extension.
This includes Debian 9 and Amazon Linux 1. I am following up to get fixes in place in both of those distributions prior to Fastly's cutoff date.
Once you get those followups started, it would be helpful to link to them. It will help to coordinate backports for other distros that are open to backporting the changes. Thanks :-)
I checked Python 2.7.5 on RHEL 7.9 with tcpdump. pip requests to pypi and pythonhosted are sending the SNI TLS extension.
@tiran the issue isn't with pip, so you don't need to test that. The issue is with setuptools. I've been testing by creating a setup.py file containing the following:
from setuptools import setup, find_packages
setup(name='sni_test', version='1.0', include_package_data=True, install_requires=['pytest'])
And then running python setup.py install --user while capturing tcp packets with tcpdump in another session. There may well be a better way to write this setup.py, but this is sufficient.
Just tested CentOS 7 ("CentOS Linux release 7.9.2009 (Core)"), which includes python-setuptools-0.9.8-7.el7.noarch. Confirmed that it does not send SNI extensions. I'd expect RHEL 7 to have similar behavior.
In order to provide visibility for this deprecation and give people a chance to test their fixes we will be implementing rolling brownouts of non-SNI support beginning later today. The mechanism will follow similar pattern to our TLS1.0/1.1 deprecation.
Support for non-SNI clients will be disabled:
| Dates | Window(s) |
|---|---|
| March 31st - April 7th | :00-:10 every hour |
| April 7th - April 14th | :00-:15 every hour |
| April 14th - April 21st | :00-:15 and :30-:40 every hour |
| April 21st - April 28th | :00-:15 and :30-:45 every hour |
| April 28th - May 3rd | :00-20 and :30-:50 every hour |
| May 3rd onward | Support for non-SNI clients disabled in our VCL |
| May 3rd - May 6th | Fastly will migrate our host, disabling non-SNI clients completely |
https://github.com/python/pypi-infra/pull/66/files Implements the rolling brownouts described above. Once merged they will be in progress and proceed as described in the timetable above.
Note transitions occur at 12:00:00 UTC
So do we know whether centos7 is affected? I see conflicting answers here:
#978 (comment) vs #978 (comment)
In testing the above PR to implement the brownouts, I confirmed that test.pypi.org and test-files.pythonhosted.org have already completely disabled non-SNI support, so using pip --index-url=https://test.pypi.org/simple/ is probably the easiest way to test/confirm.
So do we know whether centos7 is affected? I see conflicting answers here:
#978 (comment) vs #978 (comment)
pip on CentOS 7 is not affected. It look like setuptools is affected, though. It didn't occur to me that setuptools does not use pip to fetch dependencies. Setuptools uses different approach download packages.
The configuration for the rolling brownouts has been merged and applied.
It didn't occur to me that setuptools does not use pip to fetch dependencies.
Older versions of setuptools don't. This changed recently, and the current versions of setuptools should be using pip for these things in some form.
When I contacted Fastly when things started breaking, they suggested I speak with my account manager, which is the beginning of a commercial, rather than a technical conversation.
Which makes sense to me.
It's very likely that keeping non-SNI things working for a certain customer would likely be something that comes with a decently large bill (there's an ongoing cost to keeping old things working, especially if they're related to security). And I imagine that the bill in this case would be large enough to justify a reprioritisation of internal resources for Fastly. :)
(this is a personal opinion and all that)
Amazon Linux 1
Time to page my old team - @iliana Could you send this to the Amazon Linux Team on this issue?
@andrewegel I dropped a link in Slack, I’ll try to take a look Monday.
AWS has published an updated setuptools package for Amazon Linux 1, so this issue should be addressed there as of today:
[ec2-user@ip-10-0-0-249 ~]$ rpm -q --changelog python27-setuptools-36.2.7-1.35.amzn1.noarch| head -n 3
* Fri Apr 02 2021 ssuryad <ssuryad@amazon.com>
- Add patch for sending SNI extensions as part of the TLS client negotiation
I've reported the issue to the Debian LTS team responsible for updating Debian 9 and received an acknowledgement, so it should be getting addressed there as well.
tip:
If you are using old enough virtualenv, you might get into trouble. pip is bundled in virtualenv. Old virtualenv = old pip. Updating virtualenv to a newer version helped me with one older python project
We have received confirmation from our CDN provider on the date for the hard disabling of SNI: May 5th, 2021 between 9am and 5pm Pacific.
This does not affect the existing timeline or brownout schedule.
Local proxy cache can be used to support old client
nginx config that i use with docker image `nginx:stable-alpine`
user nginx; pid /run/nginx.pid; worker_processes auto; worker_rlimit_nofile 65535;events {
multi_accept on;
worker_connections 65535;
}http {
charset utf-8;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
server_tokens off;
log_not_found off;
types_hash_max_size 2048;
client_max_body_size 16M;# MIME include mime.types; default_type application/octet-stream; # For container environment set_real_ip_from 172.0.0.0/8; proxy_cache_path /data/cache levels=1:2 keys_zone=STATIC:10m inactive=7d max_size=1g; server { listen 80; listen [::]:80; # security # security headers add_header X-Frame-Options "SAMEORIGIN" always; add_header X-XSS-Protection "1; mode=block" always; add_header X-Content-Type-Options "nosniff" always; add_header Referrer-Policy "no-referrer-when-downgrade" always; add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always; # gzip gzip on; gzip_vary on; gzip_proxied any; gzip_comp_level 6; gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml; # enable sni https://github.com/pypa/pypi-support/issues/978 proxy_ssl_server_name on; # reverse proxy location /packages/ { proxy_pass https://files.pythonhosted.org/packages/; proxy_buffering on; proxy_cache STATIC; proxy_cache_valid 200 7d; proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504; } location / { proxy_pass https://pypi.org/; proxy_set_header Accept-Encoding ""; proxy_buffering on; proxy_cache STATIC; proxy_cache_valid 200 1s; proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504; sub_filter_once off; sub_filter '<a href="https://files.pythonhosted.org/packages/' '<a href="$scheme://$host/packages/'; } }}
then set PIP_INDEX_URL to the proxy cache
We are getting this error in python 2.7.18 as well:
Could not fetch URL https://pypi.python.org/simple/requests/: HTTP Error 403: [[[!!! BREAKING CHANGE !!!]]] Support for clients that do not support Server Name Indication is temporarily disabled and will be permanently deprecated soon. See https://status.python.org/incidents/hzmjhqsdjqgb and #978 [[[!!! END BREAKING CHANGE !!!]]]
Will skip URL https://pypi.python.org/simple/requests/ when looking for download links for requests
We migrated to 2.7.18 from an earlier 2.7.x version in the hope that version > 2.7.9 will not be impacted. Is this also not supposed to work?
@azitabh what command are you running to receive that error? what utility and version of said utility?
Please provide versions for Python and tools that are experiencing errors. We cannot assist without this information.
Just to be sure, as our Python upgrade is still ongoing and will take some time, if we have RHEL 7.6 we should be ok for now because python2 -c "import ssl; print(ssl.HAS_SNI)" reports True?
And can we still install local packages via pip, if we would get problems later?
@arcukwasshka at this point if installs are working, you're in the clear. as of ~14 hours ago SNI has been disabled via our configuration, at some point soon it will be completely disabled, but the effect will be the same.
@arcukwasshka at this point if installs are working, you're in the clear. as of ~14 hours ago SNI has been disabled via our configuration, at some point soon it will be completely disabled, but the effect will be the same.
Thanks for the quick reply!
I was able to upgrade a package in our test env (pip showed that Downloading https://files.pythonhosted.org/packages...) and all I got was the usual Python 2.7 deprecation warning.
So it seems we are in the clear.
I'm using pip with python 2.7.6, and I'm getting this issue. Just wanted to check if I'm going to be permanently disabled from using pip with my current tools or is this just temporary? Many thanks in advance.
@igorop this change is now permanent as of 2021-05-04T00:00:00 UTC. You may benefit from upgrading to the latest Python 2.7 release which has SNI support.
Thank you for the quick response @ewdurbin . It's not that simple for me to upgrade the python version because of other dependencies, although I intended to upgrade over time. Is there no other possible workaround?
I upgraded to 2.7.14 but I still get the same issue:
I actually had a bad installation of pip which caused the issue, its working now with python 2.7.14
I upgraded to 2.7.14 but I still get the same issue:
Please provide error messages as text. Images are not accessible or searchable.
Just wanted to add that I've seen this issue just now when running behind a proxy. The proxy is Nexus, and is running on CentOS7 and Java 8. Seems like upgrading the JVM from Java 8 would fix it (https://stackoverflow.com/a/58953309/741316). I can imagine a similar story for those behind an Artifactory based proxy running on Java 8 (unverified).
Edit: Slightly uncomfortable story... nexus doesn't support Java >8 (needed for the fix). I tried a quick and dirty workaround to setup an nginx proxy and configuring my nexus to point to that... it is important to now include the following in the nginx config:
proxy_ssl_name pypi.org;
proxy_ssl_server_name on;
I once again am having problems with pypi.python.org. However, it does not look like an SNI issue that I observed in March. This is from a yocto build of python-protobuf-native using Python 2.6.13. The error I see now is (Couldn't find index page for 'six')
Installed /home2/jay/work/tpm/sc20_linux/poky/build/tmp-glibc/work/x86_64-linux/python-protobuf-native/3.3.0-r0/image/home2/jay/work/tpm/sc20_linux/poky/build/tmp-glibc/work/x86_64-linux/python-protobuf-native/3.3.0-r0/recipe-sysroot-native/usr/lib/python2.7/site-packages/protobuf-3.3.0-py2.7.egg
Processing dependencies for protobuf==3.3.0
Searching for six>=1.9
Reading https://pypi.python.org/simple/six/
Couldn't find index page for 'six' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
No local packages or working download links found for six>=1.9
error: Could not find suitable distribution for Requirement.parse('six>=1.9')
ERROR: python setup.py install execution failed.
Any ideas on how to fix this?
Any ideas on how to fix this?
Please reach out to the support channels of wherever you're getting your Python installation/distribution (eg: Amazon for Amazon Linux stuff, RedHat for RHEL stuff etc). Those folks are in a better position to diagnose your issue than we are.
Thanks. I think I have a work around to avoid the issue.
I think the issue is that python setup is trying to download and install python-six to satisfy a dependency of python-protobuf, and for some reason this no longer works.
I have a work around where I add a yocto dependency on python-six to the python-protobuf recipe, so python-six is fetched using yocto's fetcher and built and installed before python-protobuf builds. This way the dependency is met by the locally installed package and avoids setup needing to download it (which is no longer working). This is a better way of doing this because the version of python-six used by python-protobuf will be the same as the version of python-six installed and it avoids internet access when using a source pre mirror.
Hello,
I am not sure if this has to do with some of the issues we have been having with our project, but we recently started having "INIT_SCRIPT_FAILURE(CLIENT_ERROR)" errors.
Image of error
Background
Our set up is using Azure Data Factory and Databricks. We are orchestrating steps in an ETL pipeline that runs different notebooks in a specific pattern. At the start of each notebook we specify a cluster that contains the following packages:
These packages are in a "whl" file and when a cluster spins up it gets the list form the "whl" file. We then saw the error (all the way at the bottom) that the wheel file was returning a non 0 read error.:
We then noticed the timeout errors when connecting to PyPI to get the projects that the Databricks cluster is using:
We started to have these issues on the 6th of this month (before that we never received the above error) and were wondering if its possible that we are being affected by this. At the top of the page it does say that Brownouts should be done by the 5th of May so we assumed that it might be over soon, but we got hit with the same error on the 8th and the 9th.
We are not using any SNI clients, but were wondering if the issue is at all related to the above. Our solution so far is to re-run the notebook and it seems to solve the issue. We checked the rest of our code, but this issue is intermittent and is resolved when we re-run the notebook.
Please let me know if you need any additional details. We just want to see if this deprecation might be causing the issue we are having.
Thank you in advance.
Azure Data Factory and Databricks
Please reach out to these vendors. They're in a better position to provide support for their infrastructure and software running in it, than the volunteers working on PyPI.
This issue is being closed and locked. Users impacted by this should direct further support requests to the channels for the platform or vendor supporting their toolchain.