Blosc/c-blosc

Maximum blosc level isn't the max (zstd)?

eggbit opened this issue · 1 comments

eggbit commented

I'm doing some experimenting and I'm noticing different results when using the same version of the zstd library (standalone vs. bloc).

Example with a 576.29 MB file:
Using zstd 1.3.8 with ZSTD_maxCLevel(): 363.53 MB
Using zstd 1.3.8 in blosc with level 9: 379.26 MB

Using BLOSC_SHUFFLE and BLOSC_NOSHUFFLE gave the same output and using BLOSC_BITSHUFFLE was actually worse (425 MB).

Any idea why blosc's max level isn't matching zstd's max level?

Thanks!

FrancescAlted commented

As fas as I see, Blosc is actually using the maximum compression level for Zstd for clevel = 9.

What is happening here that Blosc splits the hole data in smaller blocks (in order to favor cache locality and maximize the use of multiple cores for speed), and that means less opportunities for finding redundancies. In order to increase the compression ratio, you can make the blocksize to be the same size than your working size.

  • 👍1