Retrying same page over and over in some cases
geekgonecrazy opened this issue · 4 comments
I was trying this out on our github repo: https://github.com/rocketchat/rocket.chat and it got stuck on a single item in history. Added some verbose logs and it was getting stuck downloading the same page over and over again. I had to modify:
https://github.com/google/triage-party/blob/master/pkg/hubbub/timeline.go#L64-L66
to:
if resp.NextPage == 0 || sp.ListOptions.Page == resp.NextPage {
break
}
sp.ListOptions.Page = resp.NextPage
The one it actually got stuck on: RocketChat/Rocket.Chat#14278 not sure why 🤔
👀 I like breaking things. :)
Seems to also get stuck downloading the same list of PR's over and over again:
I0916 21:27:59.099728 1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:27:59.099814 1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:27:59.941729 1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:27:59.942050 1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:28:00.845414 1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:28:00.845505 1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:28:02.205749 1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:28:02.205990 1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:28:03.211754 1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:28:03.211843 1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:28:04.190577 1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:28:04.190703 1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:28:05.277635 1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:28:05.277678 1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
Adding the same sort of logic to the pull_requests.go seemed to do the trick also. I don't know what kind of info might be missed because of this. Just treating the symptom basically 🙈
I have a similar but I don't know if the same problem when trying to fetch data for apache/airflow repository.
I get the impression that the program goes into an infinite loop and can never get all the data.
Yup seems to try getting the same page over and over blowing through the api limits repeatedly. It wasn’t until I added the check both places that it started working.
I’ll open a PR with the changes I made