Memory management and full datafile downloads
neilsmurphy opened this issue · 1 comments
neilsmurphy commented
Currently the downloads can fail due to memory isssues when downloading some of the larger files.
Need to implement method to batch downloads on date columns and then also on a sub column, likely tickers?
Then if this is done natural to progress to downloading all files for initial installation.
See if there's an existing library for managing batch size. Or perhaps time the batches and adjust size accordingly.
neilsmurphy commented
Created downloading for large files in zip and then creating new database. This is not in the repo. Try to have auto update happen if possilbe with a auto running script.