iclab/centinel

Replace bzip2 with zstd or lz4

Opened this issue · 0 comments

zackw commented

bzip2 compression produces more compact files on disk than gzip, but it's also very slow both to compress and to decompress. We should consider switching to a newer compression algorithm. Fast decompression should be a priority. zstd is somewhat faster than bzip2 (especially at decompression) for equivalent compressed sizes, and lz4 is much faster than bzip2 but only about as good as gzip when it comes to compressed sizes.