Legit-Labs/legitify

"panic: send on closed channel" error when running 0.2.5

ashvince-tide opened this issue · 5 comments

ashvince-tide commented

TL;DR

Running Legitify against our GitHub organisation with 800 repositories fails with an error message saying "panic: send on closed channel" and a stack trace (pasted below).

The bottom progress bar shows "secondary rate limit" and a counter going up to 60 seconds beforehand.

Expected behavior

I would expect to see some output from Legitify showing the analysis of our GitHub organisation

Observed behavior

The error output shown below

Version

0.2.5

On which operating system are you using legitify?

Mac OS

Relevant log output

panic: send on closed channel

goroutine 817 [running]:
github.com/Legit-Labs/legitify/internal/collectors.(*BaseCollector).CollectDataWithContext(...)
	/home/runner/work/legitify/legitify/internal/collectors/collector_utils.go:58
github.com/Legit-Labs/legitify/internal/collectors/github.(*repositoryCollector).collectRepository(0xc0005487d0, 0x2f6d6f632e627568?, {0xc000332360, 0xc}, 0xc000314200)
	/home/runner/work/legitify/legitify/internal/collectors/github/repository_collector.go:233 +0x413
github.com/Legit-Labs/legitify/internal/collectors/github.(*repositoryCollector).collectRepositories.func1.1()
	/home/runner/work/legitify/legitify/internal/collectors/github/repository_collector.go:210 +0x271
github.com/Legit-Labs/legitify/internal/common/group_waiter.(*GroupWaiter).Do.func1()
	/home/runner/work/legitify/legitify/internal/common/group_waiter/group_waiter.go:25 +0x5b
created by github.com/Legit-Labs/legitify/internal/common/group_waiter.(*GroupWaiter).Do
	/home/runner/work/legitify/legitify/internal/common/group_waiter/group_waiter.go:23 +0x8a

Additional information

No response

gal-legit commented

Hi @ashvince-tide, thank you for reporting the issue.
I failed to reproduce it locally.
Could you please share the error.log (or a redacted version)? That'll help us understand the flow better.

Also, could you please share the following:

  1. Does the problem continuously reproduce?
  2. Was the secondary rate limit bar completed (and disappeared) before the panic?
  3. How many repositories did the progress bar show before the panic?
  4. Did you run the analyze command with the --org flag?

thanks

ashvince-tide commented
  1. Yes, seems to be consistent
  2. Yes, It counts up to 59 seconds, then disappears.
  3. It seems to consistently get to 49 out of 829 repositories
  4. Yes
2023/03/15 17:42:58 2023/03/15 17:42:58 facing secondary rate limit with request: https://api.github.com/repos/**/**/vulnerability-alerts. sleeping until: 2023-03-15 17:42:59.986957 +0000 GMT m=+5.786286639
2023/03/15 17:42:59 2023/03/15 17:42:59 attempt 1/5 failed: collect repositories for ** with err: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"
2023/03/15 17:43:00 2023/03/15 17:43:00 facing secondary rate limit with request: https://api.github.com/repos/**/**/vulnerability-alerts. sleeping until: 2023-03-15 17:43:01.192483 +0000 GMT m=+6.991792921
2023/03/15 17:43:00 2023/03/15 17:43:00 attempt 2/5 failed: collect repositories for ** with err: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"
2023/03/15 17:43:01 2023/03/15 17:43:01 attempt 3/5 failed: collect repositories for ** with err: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"
2023/03/15 17:43:01 2023/03/15 17:43:01 attempt 4/5 failed: collect repositories for ** with err: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"
2023/03/15 17:43:01 2023/03/15 17:43:01 facing secondary rate limit with request: https://api.github.com/repos/**/**/vulnerability-alerts. sleeping until: 2023-03-15 17:44:01.331969 +0000 GMT m=+67.131260080
2023/03/15 17:43:01 2023/03/15 17:43:01 attempt 5/5 failed: collect repositories for ** with err: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"
2023/03/15 17:43:01 2023/03/15 17:43:01 all 5 attempts failed (collect repositories for **) with err: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"
2023/03/15 17:43:01 2023/03/15 17:43:01 BUG: closing bar repository although it is not completed. please report this issue to legitify repository.
noamd-legit commented

Hi @ashvince-tide! Could you test this fix #175?

noamd-legit commented

The fix is merged to the main branch, closing this issue for now.
Please let us know if it is not solved :)

  • 👍1
  • 🎉1