Error occured: pagination is limited for this resource
ktsaou opened this issue · 16 comments
Hi,
Thank you for this excellent tool !
On popular repos I get this:
It happens on elastic/elasticsearch, kubernetes/kubernetes and generally all repos with more that 40k stars.
Yes, I see this error too. The error comes from GitHub API. I guess that there is a limitation to the number of API calls for a certain repo. I'm not sure how to solve it though...
I guess a delay and retry would solve this.
We could have a "Continue" or "Retry" button on this modal.
Keep in mind that the project has a couple key limitations:
-
It should continue where it stopped. So, you add a project. After a few days you want it updated. Now it starts from the beginning. Ideally it should be able to continue from the point it left. This would save a lot of time for big projects.
-
Data should be saved in browser local storage, so that a refresh should just bring the last view up.
-
We should be able to remove projects.
Using the above, the project would perfectly work on any size of projects.
That's a very good idea! Do you think you can try to implement it and submit a PR?
That's a very good idea! Do you think you can try to implement it and submit a PR?
Unfortunately I won't be able to help. I would love to, but I don't have the time to do it.
Got it. I'll try to work on it when I have time. Let's keep it open for now and I'll see if I can find time soon
This is another case that a backend can help with.
Instead of fetching all the stars every time, you could fetch only new stars for tracked projects and dump them into a some kind of database, an RRD file or maybe into the storage of an existing graphing solution like Graphite (I think their backend is called Carbon).
@omry Yes I agree. BTW you're more than welcome to join the project and contribute! Plz let me know if you're interested
Thanks for the offer @seladb, unfortunately I don't have cycles for this. all my time goes into my own project, Hydra (http://hydra.cc), also my web development skills are a bit outdated, last time I did anything there was over 10 years ago :).
Sure @omry , I'm also not really a web developer and I'm learning React as I go 😄
Anyway thanks for the great feedback and ideas, I'll try to implement whatever I can in the new re-write
The issue seems to be that the GitHub API is limited to 400 pages aka 40k stars. It should be possible to support repos with 80k stars by checking the first 40k stars and then checking the last 40k.
For repos above 80k stars, instead of showing an error the middle can be left empty and shown to the user as a straight line.
@probablykasper this might solve it, however loading all that data to the browser will take a lot of time and memory. There're also a performance issues with ApexCharts when loading too much data.
The ultimate solution for repos with a massive amount of stars is probably to add a backend that loads the data (hopefully not in real time) and then serve a sample of this data to the client.
@seladb Sure, but you would still have to use the same method for fetching the data on a server. For 80k repos, you can also fetch the first and last 40k in parallel.
Maybe Chart.js has better performance?
yes, that makes sense. Its' worth trying Chart.js and see if it performs better. If you're interested I invite you to try implementing your ideas, those would be very nice additions to this tool!
@seladb Got other priorities for the foreseeable future, but might try at some point