JSONDecodeError, HTML Error Code 429 Rate Limit
Closed this issue · 3 comments
Hey, my new bot wont work anymore. After about 10 hours of matchmaking it returns a JSONDecodeError and the HTML error code 429. I already read #501 and #510 and already incresead the challenge timeout to 2 minutes but even after pausing the bot for some time i still get this error:
DEBUG Starting new HTTPS connection (1): lichess.org:443 connectionpool.py:1003
DEBUG https://lichess.org:443 "GET /api/account HTTP/1.1" 200 None connectionpool.py:456
INFO Welcome uchessBOT! lichess-bot.py:792
INFO You're now connected to https://lichess.org/ and awaiting challenges. lichess-bot.py:144
[03/22/23 08:13:47] DEBUG https://lichess.org:443 "GET /api/account/playing HTTP/1.1" 200 17 connectionpool.py:456
[03/22/23 08:15:49] INFO Challenging a random bot matchmaking.py:156
INFO Seeking blitz game with opponent rating in [1637, 2037] ... matchmaking.py:111
DEBUG Resetting dropped connection: lichess.org connectionpool.py:273
DEBUG https://lichess.org:443 "GET /api/bot/online HTTP/1.1" 200 None connectionpool.py:456
[03/22/23 08:15:51] DEBUG https://lichess.org:443 "GET /api/user/Lynx_BOT HTTP/1.1" 200 None connectionpool.py:456
INFO Will challenge Lynx_BOT for a standard game. matchmaking.py:159
DEBUG https://lichess.org:443 "POST /api/challenge/Lynx_BOT HTTP/1.1" 429 35 connectionpool.py:456
WARNING Endpoint /api/challenge/{} is rate limited. Waiting 60 seconds until next request. lichess.py:145
ERROR Could not create challenge matchmaking.py:68
Traceback (most recent call last):
File "/home/jan/informatik/py/uchess/lichess-bot/venv/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jan/informatik/py/uchess/lichess-bot/matchmaking.py", line 61, in create_challenge
response = self.li.challenge(username, params)
File "/home/jan/informatik/py/uchess/lichess-bot/lichess.py", line 230, in challenge
return self.api_post("challenge", username, payload=payload, raise_for_status=False)
File "/home/jan/informatik/py/uchess/lichess-bot/venv/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
File "/home/jan/informatik/py/uchess/lichess-bot/lichess.py", line 134, in api_post
json_response: JSON_REPLY_TYPE = response.json()
File "/home/jan/informatik/py/uchess/lichess-bot/venv/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
INFO Challenge id is None.
This is my matchmaking config:
matchmaking:
allow_matchmaking: true
challenge_variant: random
challenge_timeout: 2
challenge_initial_time:
- 300
- 180
challenge_increment:
- 1
- 0
- 2
- 3
opponent_rating_difference: 200
opponent_allow_tos_violation: true
challenge_mode: rated
delay_after_decline: none
block_list: []
challenge_days:
- null
opponent_min_rating: 600
opponent_max_rating: 4000
There are daily limits as well as rate limits for challenges. I think the maximum number of challenges in a day is 400. This corresponds to a matchmaking: challenge_timeout of 3.6 minutes. Plus, every declined or ignored challenge is followed by another challenge after waiting 1 minute. So it's reasonable that your bot hit that limit after 10 hours. Setting challenge_timeout to 5 minutes should allow your bot to issue challenges without hitting daily limits.
As of now, I can see a few possible changes to lichess-bot that could alleviate this for all users:
- Make the absolute minimum time between calls to the lichess challenge endpoint 4 minutes (or, 216 seconds to get exactly 400 challenges per day). This would be in effect even if a challenge is declined or cancelled.
- Keep a list of Timers, one for every call to the challenge endpoint and set to 24 hours. If the number of unexpired timers exceeds 400, stop creating new challenges until enough of them expire.
- Same as 2, but instead of hardcoding 400 daily challenges, a 429 response from the challenge endpoint will cause matchmaking to pause until the oldest timer expires.
@AttackingOrDefending Thoughts?
The problem with 2 and 3 is that the bot will use matchmaking for some hours (e.g. 10) and then not use it for the rest of the day (14 hours in this example). A way to make sure that the bot always uses matchmaking is to keep a list of timers and if they exceed 100 multiply the minimum time by 5 (also for declined or cancelled challenges), which guarantees a minimum wait time of 5 minutes. When they drop below 90 divide it by 5. We could also add 3 as a fail-safe.
@jthuermann Please try PR #679 and let us know if you still run into rate limiting.