Support gzipped files for historical data out-of-the-box
petedmarsh opened this issue · 3 comments
At the moment the FlumineHistoricalGeneratorStream does not support gzipped files out-of-the-box:
The performance docs mention smart_open which would support gzipped local files as well as S3 etc. This is an option but it's perhaps offering too many features out-of-the-box,
It would be fairly simple to add in gzip file support with some simple heuristics using Python built-ins:
https://stackoverflow.com/questions/3703276/how-to-tell-if-a-file-is-gzip-compressed
I will happily implement either one (I suggest the later using Python standard library stuff) - do you have a preferance?
There is a PR open for this, I don't like it, smart open does it all.
Cool, I can open another using smart_open
It won't be merged