hentai-chan/hentai

Library not working due to poor request

JengRung opened this issue · 3 comments

Describe the Bug

Getting the 'too many 503 error responses' error when calling the package

Steps To Reproduce

from hentai import Hentai

doujin = Hentai(177013)
print(Hentai.exists(doujin.id))

System Information

| OS Name | Python Version | App Version |
|:---------- --:|:-----------------:|:-----------:|
| Windows | 3.10.2 | 3.2.10 |

Error message & warnings

Traceback (most recent call last):
  File "C:\Python310\lib\site-packages\requests\adapters.py", line 440, in send
    resp = conn.urlopen(
  File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 876, in urlopen
    return self.urlopen(
  File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 876, in urlopen
    return self.urlopen(
  File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 876, in urlopen
    return self.urlopen(
  [Previous line repeated 2 more times]
  File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 866, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "C:\Python310\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nhentai.net', port=443): Max retries exceeded with url: /api/gallery/177013 (Caused by ResponseError('too many 503 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\xxx\Desktop\hentai scrping\hentai\test.py", line 3, in <module>
    doujin = Hentai(177013)
  File "C:\Python310\lib\site-packages\hentai\hentai.py", line 531, in __init__
    self.__response = self.handler.get(self.api)
  File "C:\Python310\lib\site-packages\hentai\hentai.py", line 479, in get
    response = self.session.get(url, timeout=self.timeout, proxies=self.proxies or getproxies(), **kwargs)
  File "C:\Python310\lib\site-packages\requests\sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "C:\Python310\lib\site-packages\requests\sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python310\lib\site-packages\requests\sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python310\lib\site-packages\requests\adapters.py", line 510, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='nhentai.net', port=443): Max retries exceeded with url: /api/gallery/177013 (Caused by ResponseError('too many 503 error responses'))

Additional context

I know that we all have a hard time scraping nhentai due to Cloudflare. Just wondering if there is a good solution now? Also is there any instruction on how to set up OAuth2 with this package?

This is a duplicate of issue #155. A developer at nhentai.net needs to implement OAuth2 first (which is something I have no control over of) so that clients (such as this hentai library) can pick up on that change and follow suit. I haven't been able to contact the site admin myself. I think they added a rule which automatically enables cloudflare protection as soon as a certain threshold is crossed which is why sometimes the API works again. Maybe if more developers try to contact them via twitter (https://twitter.com/nhentaiOfficial) a solution can be found that works for everyone.

Update: There's a potential fix in #163 on its way, let's see how this plays out.

Relevant comment about this issue:

#163 (comment)