download-directory/download-directory.github.io

Deal with rate limiting

Opened this issue · 2 comments

Rate limiting is only implemented on the first load:

if (response.headers.get('X-RateLimit-Remaining') === '0') {

I think that if you hit it while downloading many files it will just continue making (and retrying) requests and eventually fail to build the zip.

Tasks:

  • Have all the fetch functions automatically use the global token if available
  • Find a proper solution to handle downloads, like halting them all when the rate is limited

I wish I could just load @octokit/rest and do it properly, but it might be huge. This tool says it's only 17KB though: https://bundlejs.com/?q=%40octokit%2Frest&treeshake=%5B*%5D&text=%22const+octokit+%3D+new+Octokit%28%29%3B%5Cn%5Cn%2F%2F+Compare%3A+https%3A%2F%2Fdocs.github.com%2Fen%2Frest%2Freference%2Frepos%2F%23list-organization-repositories%5Cnoctokit.rest.repos%5Cn++.listForOrg%28%7B%5Cn++++org%3A+%5C%22octokit%5C%22%2C%5Cn++++type%3A+%5C%22public%5C%22%2C%5Cn++%7D%29%5Cn++.then%28%28%7B+data+%7D%29+%3D%3E+%7B%5Cn++++%2F%2F+handle+data%5Cn++%7D%29%3B%22

Current situation, trying with:

https://github.com/phracker/MacOSX-SDKs/tree/master/MacOSX10.14.sdk
Screenshot 6