guidopetri/chess-pipeline

Progress bar refactor

Closed this issue · 1 comments

Currently, the progress bar indication on Luigi is based off of "time of day played". This is semi-useful in that it gives some indication of how far we've analyzed/downloaded games, but it's not the best, because if someone plays 100 games in a day and 1 game in the rest of the week, it looks like we're not making a lot of progress (since we're not progressing through time as much).

It would be better to have a counter that is based off of "which number game are we analyzing/downloading currently". However, I think this requires another lichess API pull - just so that we can count how many games we have to parse. Alternatively, we could pull each game from the lichess API individually (or in batches), then count them and then analyze them. Keeping the games in memory would be a drag though, so it would have to be in its own Task that writes to (several) files. Is this really a better idea?

This was implemented just by using another request (which will hopefully be lightweight). 5499d28