runatlantis/atlantis

Atlantis crashes when hitting GitHub rate limit

adubeniuk opened this issue · 0 comments

adubeniuk commented

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

GitHub user access token requests are limited to 5,000 requests per hour and per authenticated user. When using one heavily-used GitHub account for Atlantis it is possible to hit a rate limit. When that happens Atlantis throws an error and crashes.

Reproduction Steps

Use GitHub account, create 5000 test requests to API then trigger any Atlantis command.

Logs

Logs 
{"level":"error","ts":"2023-09-04T15:08:54.599Z","caller":"vcs/instrumented_client.go:142","msg":"Unable to react to comment, error: POST https://api.github.com/repos/[REDACTED]/issues/comments/[REDACTED]/reactions: 403 API rate limit exceeded for user ID [REDACTED]. [rate reset in 34s]","json":{},"stacktrace":"github.com/runatlantis/atlantis/server/events/vcs.(*InstrumentedClient).ReactToComment\n\tgithub.com/runatlantis/atlantis/server/events/vcs/instrumented_client.go:142\ngithub.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).ReactToComment\n\tgithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:68\ngithub.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleCommentEvent\n\tgithub.com/runatlantis/atlantis/server/controllers/events/events_controller.go:570\ngithub.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).HandleGithubCommentEvent\n\tgithub.com/runatlantis/atlantis/server/controllers/events/events_controller.go:318\ngithub.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleGithubPost\n\tgithub.com/runatlantis/atlantis/server/controllers/events/events_controller.go:177\ngithub.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).Post\n\tgithub.com/runatlantis/atlantis/server/controllers/events/events_controller.go:104\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2122\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.0/mux.go:210\ngithub.com/urfave/negroni/v3.Wrap.func1\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:59\ngithub.com/urfave/negroni/v3.HandlerFunc.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:33\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP\n\tgithub.com/runatlantis/atlantis/server/middleware.go:70\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/urfave/negroni/v3.(*Recovery).ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/recovery.go:210\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/urfave/negroni/v3.(*Negroni).ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:111\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2936\nnet/http.(*conn).serve\n\tnet/http/server.go:1995"}
{"level":"warn","ts":"2023-09-04T15:08:54.599Z","caller":"events/events_controller.go:572","msg":"Failed to react to comment: POST https://api.github.com/repos/[REDACTED]/issues/comments/[REDACTED]/reactions: 403 API rate limit exceeded for user ID [REDACTED]. [rate reset in 34s]","json":{"gh-request-id":"X-Github-Delivery=[REDACTED]"},"stacktrace":"github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleCommentEvent\n\tgithub.com/runatlantis/atlantis/server/controllers/events/events_controller.go:572\ngithub.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).HandleGithubCommentEvent\n\tgithub.com/runatlantis/atlantis/server/controllers/events/events_controller.go:318\ngithub.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleGithubPost\n\tgithub.com/runatlantis/atlantis/server/controllers/events/events_controller.go:177\ngithub.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).Post\n\tgithub.com/runatlantis/atlantis/server/controllers/events/events_controller.go:104\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2122\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.0/mux.go:210\ngithub.com/urfave/negroni/v3.Wrap.func1\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:59\ngithub.com/urfave/negroni/v3.HandlerFunc.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:33\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP\n\tgithub.com/runatlantis/atlantis/server/middleware.go:70\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/urfave/negroni/v3.(*Recovery).ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/recovery.go:210\ngithub.com/urfave/negroni/v3.middleware.ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngithub.com/urfave/negroni/v3.(*Negroni).ServeHTTP\n\tgithub.com/urfave/negroni/v3@v3.0.0/negroni.go:111\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2936\nnet/http.(*conn).serve\n\tnet/http/server.go:1995"}
{"level":"error","ts":"2023-09-04T15:08:54.599Z","caller":"vcs/instrumented_client.go:72","msg":"Unable to get pull number for repo, error: GET https://api.github.com/repos/[REDACTED]/pulls/3374: 403 API rate limit of 5000 still exceeded until 2023-09-04 15:09:29 +0000 UTC, not making remote request. [rate reset in 34s]","json":{"repository":"[REDACTED]","pull-num":"3374"},"stacktrace":"github.com/runatlantis/atlantis/server/events/vcs.(*InstrumentedGithubClient).GetPullRequest\n\tgithub.com/runatlantis/atlantis/server/events/vcs/instrumented_client.go:72\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).getGithubData\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:311\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).ensureValidRepoMetadata\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:371\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:263"}
{"level":"error","ts":"2023-09-04T15:08:54.600Z","caller":"events/command_runner.go:387","msg":"making pull request API call to GitHub: GET https://api.github.com/repos/[REDACTED]/pulls/3374: 403 API rate limit of 5000 still exceeded until 2023-09-04 15:09:29 +0000 UTC, not making remote request. [rate reset in 34s]","json":{"repo":"[REDACTED]","pull":"3374"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).ensureValidRepoMetadata\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:387\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:263"}
{"level":"error","ts":"2023-09-04T15:08:54.600Z","caller":"events/command_runner.go:429","msg":"PANIC: runtime error: invalid memory address or nil pointer dereference\nruntime/panic.go:260 (0x44e11c)\nruntime/signal_unix.go:837 (0x44e0ec)\ngithub.com/runatlantis/atlantis/server/events/command_runner.go:388 (0xf70451)\ngithub.com/runatlantis/atlantis/server/events/command_runner.go:263 (0xf6dfb7)\nruntime/asm_amd64.s:1598 (0x46b2e0)\n","json":{"repo":"[REDACTED]","pull":"3374"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:429\nruntime.gopanic\n\truntime/panic.go:890\nruntime.panicmem\n\truntime/panic.go:260\nruntime.sigpanic\n\truntime/signal_unix.go:837\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).ensureValidRepoMetadata\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:388\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:263"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xf70e83]

goroutine 100 [running]:
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics(0xc0006024e0, {{0xc000430fe0, 0x19}, {0xc000430fe0, 0x9}, {0xc000430fea, 0xf}, {0xc000088460, 0x64}, {0xc00020d680, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/events/command_runner.go:430 +0x203
panic({0x1173080, 0x1dae110})
	runtime/panic.go:890 +0x263
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).ensureValidRepoMetadata(_, {{0xc000430fe0, 0x19}, {0xc000430fe0, 0x9}, {0xc000430fea, 0xf}, {0xc000088460, 0x64}, {0xc00020d680, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/events/command_runner.go:388 +0x871
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand(0xc0006024e0, {{0xc000430fe0, 0x19}, {0xc000430fe0, 0x9}, {0xc000430fea, 0xf}, {0xc000088460, 0x64}, {0xc00020d680, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/events/command_runner.go:263 +0x718
created by github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleCommentEvent
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:594 +0x750

Environment details

If not already included, please provide the following:

  • Atlantis version: v0.25.0
  • Deployment method: eks
  • If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: yes
  • Atlantis flags:

Atlantis server-side config file:

atlantis-url: [REDACTED]
automerge: false
autoplan-file-list:  [REDACTED]
disable-autoplan: false
disable-repo-locking: true
enable-diff-markdown-format: true
hide-prev-plan-comments: true
repo-allowlist:  [REDACTED]
repo-config: /etc/atlantis/repos.yaml
silence-no-projects: true
silence-vcs-status-no-plans: true
vcs-status-name: "atlantis-dev"
executable-name: "atlantis-dev"
write-git-creds: true

Any other information you can provide about the environment/deployment (efs/nfs, aws/gcp, k8s/fargate, etc)

Additional Context