sunaku/dasht

GitHub rate limiting 403 on dasht-docsets-install

AdamWagner opened this issue · 3 comments

AdamWagner commented

Love what you've done with dasht - great concept!

Unfortunately, I haven't been able to test-drive it yet due to dasht-docsets-install resulting in a 403 due to a RateLimit issue on the feeds url. Here's the full error:

❯ dasht-docsets-install
  HTTP/1.1 403 Forbidden
  Server: GitHub.com
  Date: Wed, 02 Jan 2019 07:47:12 GMT
  Content-Type: application/json; charset=utf-8
  Content-Length: 247
  Status: 403 Forbidden
  X-RateLimit-Limit: 60
  X-RateLimit-Remaining: 0
  X-RateLimit-Reset: 1546415757
  X-GitHub-Media-Type: github.v3; format=json
  Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type
  Access-Control-Allow-Origin: *
  Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
  X-Frame-Options: deny
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
  Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
  Content-Security-Policy: default-src 'none'
  X-GitHub-Request-Id: EC95:6FB3:2E874DD:76A2D0E:5C2C6C80
https://api.github.com/repos/Kapeli/feeds/git/refs/heads/master:
2019-01-02 02:47:12 ERROR 403: Forbidden.
  HTTP/1.1 403 Forbidden
  Server: GitHub.com
  Date: Wed, 02 Jan 2019 07:47:12 GMT
  Content-Type: application/json; charset=utf-8
  Content-Length: 247
  Status: 403 Forbidden
  X-RateLimit-Limit: 60
  X-RateLimit-Remaining: 0
  X-RateLimit-Reset: 1546415757
  X-GitHub-Media-Type: github.v3; format=json
  Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type
  Access-Control-Allow-Origin: *
  Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
  X-Frame-Options: deny
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
  Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
  Content-Security-Policy: default-src 'none'
  X-GitHub-Request-Id: EC9A:6FB3:2E874E1:76A2D1F:5C2C6C80
https://api.github.com/repos/Kapeli/feeds/git/trees/%7Brefrefs/heads/masternode_idMDM6UmVmMTEyMzkyNzk6bWFzdGVyurlhttps//api.github.com/repos/Kapeli/feeds/git/refs/heads/masterobject%7Bf6640fea1ab1e9bd1544722cf3ac0ee1e9a7416ftypecommiturlhttps//api.github.com/repos/Kapeli/feeds/git/commits/f6640fea1ab1e9bd1544722cf3ac0ee1e9a7416f%7D%7D:
2019-01-02 02:47:12 ERROR 403: Forbidden.

The output includes the same block of text twice, although it's the result of running dasht-docsets-install only once.

Running on macOS Mojave in fish shell. Have tested in bash with the same result.

sunaku commented

Possible workarounds:

  • Retry from a different IP address (VPN or ISP) that hasn't sent so much traffic to GitHub API service.
  • Inherit already installed docsets from Dash for macOS by setting the DASHT_DOCSETS_DIR envar.
  • Manually download the docsets you're interested in from the list at https://github.com/Kapeli/feeds
    • You'll find *.tgz links near the top of the file, which you can then download from your browser.
    • Run dasht-docsets-extract *.tgz on the downloaded files to install them into your dasht area.

A permanent solution:

  • Add support for specifying a GitHub access token in API requests sent by dasht to avoid rate limiting.
sunaku commented

Closing due to lack of response and because the issue seems to be beyond the scope of this project. 👮‍♂️

eikendev commented

In case anybody else faces the same problem and lands here. I had set an empty User-Agent in my wget configuration (meaning no User-Agent would be sent along the request at all). This caused dasht to fail to download anything via the API, always resulting in Forbidden. When I removed the setting, dasht worked fine again.

  • ❤️1