we need some lazy loading strategy for addins data
Closed this issue · 4 comments
There are tons of data for 2017 and 2018 since we have been running it for a full year+. Clicking on that filter basically means waiting for 5-10 seconds before data comes back, and that's already filtered down to just that year. We need to stream it into the browser, with a progress bar on top or similar.
Working on a possible solution for this... It uses mongodb's aggregate to pull only the relevant info when each chart is requested (for example, count of each distinct plugin, filtered by year for the first chart). Response times for first 2 charts is under 50ms currently. Still need to work out something for the AddinManager details.
@ksobon, I went ahead and finished up a solution for lazy-loading the data for each chart. I believe it also covers some of the other issues you mentioned last week (i.e. making changes to a chart higher up should affect the data shown in charts below it). In essence, the goal was to bring aggregation and grouping as close to the database as possible, either in aggregate pipelines in mongodb, or in the controllers on the server. This results in less data being transferred via HTTP calls, which seems to be our bottleneck.
Check it out here: master...addins-loadtime-aggregate
I believe there were some other items related to the Addins view that you had mentioned, but they're escaping me right now. Let me know if there are additional items I can tackle before doing a PR.
I will be in on Monday to review it. If you want to do a PR I can review it next week.