github:pull and github2:pull indexing very slow and not correctly enriching
Closed this issue · 2 comments
I'm running grimorielab with docker compose, when I try to use github:pull and github2:pull in setup.cfg it takes several hours to get the indexes in elastisearch, and I can't use the enirched indexes in kibana.
The rest of the github settings work much, much more quickly and working well without other configurations in kibana.
Have been looking around but haven't seen anyone with the issue, happy for any help.
The time depends on the repositories analyzed and the GitHub API rate limits.
In the case of pull requests, it is necessary to execute one API request per PR to obtain specific information like changed_files. For the case of issues, one call to /issues retrieves all the information for multiple issues.
Note: GitHub's API considers every pull request an issue. With GrimoiraLab you can analyze pull requests with the issues index and the filter in Kibana/Elastic issue_pull_request=True, but it doesn't provide detailed information about PRs.
Hey @jjmerchante
Thank you for the explanation! I thought something might be wrong since i was even using multiple github tokens from different user. I don't know why the enriching didn't work the first time but when I tried it again there were no problems.