Rate of download
nick-torenvliet opened this issue · 6 comments
Benny,
Thanks for your code.
My understanding is that yahoo will only accept 2000 requests on its API per hour from an ip authenticated client.
I would like to get a comprehensive list of available ticker symbols. I feel like slowing down your script might work.
What part of your code would a I have tweak in order to bring down the number of requests your script makes to 2000 per hour?
Am I on the right path here?
I never heard about this 2000 request limit. Can you provide a source? Also, this piece of software does not use an official API. It uses the html interface for humans, so API restrictions do not apply but some unknown (to me) rate limiting/abuse prevention algorithm might be in play.
At the moment the program waits 5 seconds after every request. So the amount of HTTP requests every hour at the moment is ~720. You can modify this value in YahooTickerDownloader.py.
These are the limits on the yql api
https://developer.yahoo.com/yql/guide/usage_info_limits.html
I understand that you are scraping the web-site and these limits don't apply.
Do you have any experience with slowing down the query rate? Does it succesfully avoid the abuse prevention/rate limiting issue?
The last time it ran it ran without any problems. If you have any problems please let me know.
Sleep time between download is now adjustable using the -s flag on master.
Please let me know if you ever got a complete set using the most recent version.