Limit Requests to Avoid 429 Exception
Danc2050 opened this issue · 5 comments
At times, the requests that karl makes to infura cause the following error:
ERROR:Karl:Exception: 429 Client Error: Too Many Requests for url: https://mainnet.infura.io/
<traceback object at 0x7f0edc560688>
If karl determined the minimum time it could send requests to infura without triggering the 429 error, it could work within that bound and still make progress while not halting the execution of the program entirely due to a 429 error.
My recommended way is to create a free account to Infura and use the generated URL.
This way it will not generated the 429 error.
Do you still think a rate limit makes sense? I want to scan as fast as possible and it's really easy to fix the issue.
Hi Luca.
First off, thanks for responding. I think your tool is really a great idea. I want to do something similar, except just parse the output of mythril output on new contracts as they are put on the blockchain, so this is a real gem of a program since it can serve as a reference point for me.
I do not know much about infura requests. However, I do remember getting a similar error with a free account and tinkering with my own program(s) too quickly. I don't know how getting a free account would be different here. Additionally, if the requests made are greater than 100,000, then a free account would not be able to sustain karl being turned on 24/7.
No need to change anything on my account. I could probably read into the code more and change it if I really had a problem.
EDIT If you are concerned it will slow down the program, have you considered using threads? One thread could scan the chain within the bound and multiple others could run mythril. I noticed that Karl seems to halt scanning when given a number of mythril addresses.
Getting a free account will unlock some limits when you use the URLs that look like https://mainnet.infura.io/v3/yourpersonalkeyhere, but indeed they are limited to 100k/day.
Implementing threads is probably a good idea. At the moment everything is single thread.
Nice. I'll check it out. As a short-term fix, I found that changing the value in this line to 5 also prevents the 429 error from happening thus far (Note: lower numbers could work as well, I haven't tried).
Line 132 in 88a19be
I'll try and come up with a multi-threaded version and do a pull request to see if you like it. It would probably only be ~10 lines added at most.
I'll close this until there's more activity.