Library not working due to poor request
JengRung opened this issue · 3 comments
Describe the Bug
Getting the 'too many 503 error responses' error when calling the package
Steps To Reproduce
from hentai import Hentai
doujin = Hentai(177013)
print(Hentai.exists(doujin.id))
System Information
| OS Name | Python Version | App Version |
|:---------- --:|:-----------------:|:-----------:|
| Windows
| 3.10.2
| 3.2.10
|
Error message & warnings
Traceback (most recent call last):
File "C:\Python310\lib\site-packages\requests\adapters.py", line 440, in send
resp = conn.urlopen(
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 876, in urlopen
return self.urlopen(
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 876, in urlopen
return self.urlopen(
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 876, in urlopen
return self.urlopen(
[Previous line repeated 2 more times]
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 866, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
File "C:\Python310\lib\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nhentai.net', port=443): Max retries exceeded with url: /api/gallery/177013 (Caused by ResponseError('too many 503 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\xxx\Desktop\hentai scrping\hentai\test.py", line 3, in <module>
doujin = Hentai(177013)
File "C:\Python310\lib\site-packages\hentai\hentai.py", line 531, in __init__
self.__response = self.handler.get(self.api)
File "C:\Python310\lib\site-packages\hentai\hentai.py", line 479, in get
response = self.session.get(url, timeout=self.timeout, proxies=self.proxies or getproxies(), **kwargs)
File "C:\Python310\lib\site-packages\requests\sessions.py", line 542, in get
return self.request('GET', url, **kwargs)
File "C:\Python310\lib\site-packages\requests\sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python310\lib\site-packages\requests\sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "C:\Python310\lib\site-packages\requests\adapters.py", line 510, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='nhentai.net', port=443): Max retries exceeded with url: /api/gallery/177013 (Caused by ResponseError('too many 503 error responses'))
Additional context
I know that we all have a hard time scraping nhentai due to Cloudflare. Just wondering if there is a good solution now? Also is there any instruction on how to set up OAuth2 with this package?
This is a duplicate of issue #155. A developer at nhentai.net needs to implement OAuth2 first (which is something I have no control over of) so that clients (such as this hentai library) can pick up on that change and follow suit. I haven't been able to contact the site admin myself. I think they added a rule which automatically enables cloudflare protection as soon as a certain threshold is crossed which is why sometimes the API works again. Maybe if more developers try to contact them via twitter (https://twitter.com/nhentaiOfficial) a solution can be found that works for everyone.
Update: There's a potential fix in #163 on its way, let's see how this plays out.
Relevant comment about this issue: