429 too many requests
Closed this issue · 7 comments
I'm using textlint with Danger in a project via GitHub Actions.
I have a lot of GitHub links in my documentation, and when textlint runs, some of the checks fail with 429 too many requests, wrongfully marking links as dead.
Is there some way to delay the time between requests that the rule makes? Or maybe some other solution?
@azu for the record, silently adding a label without commenting means the person who posted the issue won't be notified. Luckily, I happened to look at this issue again today.
Not entirely sure what kind of info you need, but here's an example of where I'm running into this issue: student-hub/acs-upb-mobile#102 (comment)
It is difficult problem.
It is reasonable that You did many request to GitHub actually and GitHub response "429 too many requests"
There is not perfect solusions.
We can do following as workaround.
- Make requests slow
textlint-rule-no-dead-link/src/no-dead-link.js
Lines 19 to 21 in 260db6a
concurrencyandintervalexperimental options may help you
- Add handling for 429 response
- suppress 429 response as options?
I thought that we do not have enough information to get reasonable resolutions.
actual request, actual response, actual limitation(not limited to GitHub.)
@azu I don't really understand the interval count, how does that work?
I would like an option to ignore 429 responses just like ignoring redirects.
@azu Hi.
I'm in for similar trouble.
My case is that a markdown file includes 20 links of GitHub.com, often responses some status 429 errors.
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/1
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/2
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/3
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/4
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/5
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/6
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/7
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/8
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/9
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/10
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/11
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/12
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/13
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/14
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/15
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/16
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/17
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/18
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/19
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/20textlint runs with --debug, the result is following as:
$ npx textlint --rule textlint-rule-no-dead-link test.md --debug
textlint:cli textlint --version: 12.0.2 +0ms
textlint:cli Running on files, stdin-filename: undefined +1ms
textlint:module-loader config Config {
textlint:module-loader configFile: undefined,
textlint:module-loader rulesBaseDirectory: undefined,
textlint:module-loader disabledRules: [],
textlint:module-loader rules: [ 'no-dead-link' ],
textlint:module-loader disabledFilterRules: [],
textlint:module-loader filterRules: [],
textlint:module-loader presets: [],
textlint:module-loader plugins: [],
textlint:module-loader pluginsConfig: {},
textlint:module-loader rulesConfig: {},
textlint:module-loader filterRulesConfig: {},
textlint:module-loader rulePaths: [],
textlint:module-loader formatterName: undefined,
textlint:module-loader quiet: false,
textlint:module-loader color: true,
textlint:module-loader cache: false,
textlint:module-loader cacheLocation: '/Users/chick-p/practice-textlint-rule-no-dead-link/.textlintcache',
textlint:module-loader ignoreFile: '/Users/chick-p/practice-textlint-rule-no-dead-link/.textlintignore'
textlint:module-loader } +0ms
textlint:module-loader Loading rules from /Users/chick-p/practice-textlint-rule-no-dead-link/node_modules/textlint-rule-no-dead-link/lib/no-dead-link.js +2ms
textlint:find-util findFiles ignore baseDir: /Users/chick-p/text-rule, normalizeIgnoreFilePath: /Users/chick-p/practice-textlint-rule-no-dead-link/.textlintignore +0ms
textlint:find-util search patterns: [ 'test.md' ] +0ms
textlint:find-util search ignore patterns: [ '**/.git/**', '**/node_modules/**' ] +0ms
textlint:engine-core Process files [ '/Users/chick-p/practice-textlint-rule-no-dead-link/test.md' ] +0ms
textlint:engine-core No Process files that are un-support extensions: [] +0ms
textlint:kernel available extensions: [ '.md', '.markdown', '.mdown', '.mkdn', '.mkd', '.mdwn', '.mkdown', '.ron', '.txt', '.text' ] +0ms
textlint:kernel use plugin: markdown +0ms
textlint:core-task no-dead-link pushReport {"message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/19 is dead. (429 too many requests)","index":0} +0ms
textlint:core-task no-dead-link pushReport {"message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/20 is dead. (429 too many requests)","index":0} +21ms
textlint:core-task no-dead-link pushReport {"message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/17 is dead. (429 too many requests)","index":0} +1ms
textlint:core-task no-dead-link pushReport {"message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/18 is dead. (429 too many requests)","index":0} +1ms
textlint:cli lint results: [{"messages":[{"type":"lint","ruleId":"no-dead-link","message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/17 is dead. (429 too many requests)","index":1113,"line":17,"column":3,"severity":2},{"type":"lint","ruleId":"no-dead-link","message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/18 is dead. (429 too many requests)","index":1183,"line":18,"column":3,"severity":2},{"type":"lint","ruleId":"no-dead-link","message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/19 is dead. (429 too many requests)","index":1253,"line":19,"column":3,"severity":2},{"type":"lint","ruleId":"no-dead-link","message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/20 is dead. (429 too many requests)","index":1323,"line":20,"column":3,"severity":2}],"filePath":"/Users/chick-p/practice-textlint-rule-no-dead-link/test.md"}] +6s
textlint:@textlint/linter-formatter formatterName: stylish +0ms
/Users/chick-p/practice-textlint-rule-no-dead-link/test.md
17:3 error https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/17 is dead. (429 too many requests) no-dead-link
18:3 error https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/18 is dead. (429 too many requests) no-dead-link
19:3 error https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/19 is dead. (429 too many requests) no-dead-link
20:3 error https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/20 is dead. (429 too many requests) no-dead-link
✖ 4 problems (4 errors, 0 warnings)429 too many requests can include Retry-After header that indicates how long to wait before making a new request.
How about waiting for the number of seconds written in this header if response includes this header?
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429
How about waiting for the number of seconds written in this header if response includes this header?
Yes. It is reasonable.
Probably, we need to add new option like maxWatingTime? maxRetryTime?.
For example, if response includes Retry-After: 6000, we want to skip the checking.
I think that is a good idea.