Checking packages from PyPI appears to be broken
Closed this issue · 15 comments
When attempting to check any package on PyPI with latest Pyroma (3.0.2), on both Windows and Linux on Python 3.7.x, an error results:
xmlrpc.client.Fault: <Fault -32500: 'HTTPTooManyRequests: The action could not be performed because there were too many requests by the client. Limit may reset in 1 seconds.'>I've tested numerous times over the period of a day, with no luck. If I were to guess, this may be due to the domain and API changes involved in the transition to Warehouse/pypi.org from the old Cheeseshop/packages.pypi.org
$ pyroma -p black
------------------------------
Checking black
Found black version 20.8b1
Traceback (most recent call last):
File "C:\Miniconda3\envs\cf-env\Scripts\pyroma-script.py", line 33, in <module>
sys.exit(load_entry_point('pyroma', 'console_scripts', 'pyroma')())
File "c:\users\c. a. m. gerlach\documents\dev\hypertool\pyroma\pyroma\__init__.py", line 101, in main
rating = run(mode, argument)
File "c:\users\c. a. m. gerlach\documents\dev\hypertool\pyroma\pyroma\__init__.py", line 120, in run
data = pypidata.get_data(argument)
File "c:\users\c. a. m. gerlach\documents\dev\hypertool\pyroma\pyroma\pypidata.py", line 37, in get_data
data = client.release_data(project, release)
File "C:\Miniconda3\envs\cf-env\lib\xmlrpc\client.py", line 1112, in __call__
return self.__send(self.__name, args)
File "C:\Miniconda3\envs\cf-env\lib\xmlrpc\client.py", line 1452, in __request
verbose=self.__verbose
File "C:\Miniconda3\envs\cf-env\lib\xmlrpc\client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "C:\Miniconda3\envs\cf-env\lib\xmlrpc\client.py", line 1170, in single_request
return self.parse_response(resp)
File "C:\Miniconda3\envs\cf-env\lib\xmlrpc\client.py", line 1342, in parse_response
return u.close()
File "C:\Miniconda3\envs\cf-env\lib\xmlrpc\client.py", line 656, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault -32500: 'HTTPTooManyRequests: The action could not be performed because there were too many requests by the client. Limit may reset in 1 seconds.'>In addition, a ResourcesWarning is generated due to the XMLRPC client not being opened in a context manager, thus leaking an unclosed socket
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=592, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('192.168.1.101', 64493), raddr=('151.101.64.223', 443)>Unless its something simple, if this was due to the PyPI API changes it likely needs a full rewrite. If that's not feasible to do in the near-term, this feature should presumably be disabled with a friendlier and more informative error message, or removed.
Still? It's been four months? I guess the XMLRPC API will stay down, and it will then need a full rewrite indeed...
@regebro Actually, a bit of Googling suggests what's going on. Per the docs, the XML-RPC API still technically works, but is deprecated in favor of the modern JSON/REST API and is in the process of being removed. As part of that, it has a "very restrictive" rate limit, which based on the error above is very likely what you're running in to here. By either slowing your rate of requests or adding backoffs and retries, you could probably squeeze a bit more more use out of the old API before you'll have to port to the new one, though not sure whether it would be worthwhile over just biting the bullet since its not clear how much longer they'll keep it online.
Yeah. I stopped using the search API, and got most of the information from the JSON API, and that seems to solve it. I'll release this tomorrow, I think.
Fantastic news, thanks!
Just to be sure what I'm doing wrong using pyroma version 3.1 on Ubuntu 18.04 (Python 3.6) :
pip3 install pyroma
...
Successfully installed docutils-0.17 pygments-2.8.1 pyroma-3.1
#pyroma guardata
------------------------------
Checking guardata
Found guardata version 0.2.2
Traceback (most recent call last):
File "/usr/local/bin/pyroma", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/pyroma/__init__.py", line 101, in main
rating = run(mode, argument)
File "/usr/local/lib/python3.6/dist-packages/pyroma/__init__.py", line 120, in run
data = pypidata.get_data(argument)
File "/usr/local/lib/python3.6/dist-packages/pyroma/pypidata.py", line 37, in get_data
data = client.release_data(project, release)
File "/usr/lib/python3.6/xmlrpc/client.py", line 1112, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python3.6/xmlrpc/client.py", line 1452, in __request
verbose=self.__verbose
File "/usr/lib/python3.6/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python3.6/xmlrpc/client.py", line 1170, in single_request
return self.parse_response(resp)
File "/usr/lib/python3.6/xmlrpc/client.py", line 1342, in parse_response
return u.close()
File "/usr/lib/python3.6/xmlrpc/client.py", line 656, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault -32500: 'HTTPTooManyRequests: The action could not be performed because there were too many requests by the client. Limit may reset in 1 seconds.'>
I guess a simple time.sleep(2) right before a request to the current API should fix this.
Or is is because the PyPi package is not updated from this source repository ? I can't find anything from v 3.1 related to this issue.
Correct me if I'm wrong, @regebro , but it appears that this issue was manually closed before the PR that implements the fix, #58 , was merged (instead of including a closes tag in that PR). Therefore, the fix is neither released nor even on master in this repo. However, @antonio-fr , if you'd like to text the fix, you can clone this repo and do hub pr checkout 58 and then pip install -e . in your current environment, which should work.
Yeah, I must have pressed the wrong button, not the first time....
@regebro No worries...I'm pretty sure every Github user has done it at least a few times, and anyone that hasn't just haven't been on the site long enough yet, haha.
Because of the significant changes I released this as 3.2b1.
Unless any major bugs shows up, I'll release 3.2 next week or so, if someone reminds me. :-)
OK, I assumed it was fixed because the issue was closed and there was a PR. But in fact, the package build was pending, so the error was still present. In a way, the real issue was the closed issue. ;)
I'll test it when the new release will be published in some days. Thanks.
I can confirm this works with version 3.2b1 (tested on Python3.8).
As a side note, I have mixed feelings about the requirements of having at least 3 owners for a package on PyPi. What about the package that are maintained by a single software editor company ?
As a side note, I have mixed feelings about the requirements of having at least 3 owners for a package on PyPi. What about the package that are maintained by a single software editor company ?
They should create a backup account for when they get locked out of the first one?
Unless any major bugs shows up, I'll release 3.2 next week or so, if someone reminds me. :-)
@regebro It has been 43 days, I think the update is safe to publish.
3.2 released