Add documentation for levels
dreamflasher opened this issue · 8 comments
At https://github.com/lz4/lz4 is a benchmark for different lz4 settings. I am interested in maximum decompression throughput, currently the best for that is "LZ4 fast 8 (v1.7.3)". How do I specify this setting with py-lz4framed?
There are some compression-level constants exposed directly from lz4, so you could do something like:
for level in range(lz4framed.LZ4F_COMPRESSION_MIN, lz4framed.LZ4F_COMPRESSION_MAX + 1):
lz4framed.compress(b'some data', level=level)I don't know which one of those corresponds to "fast 8" - it's got to say that somewhere in the lz4 documentation (or otherwise you could ask there). Maybe it is just 8 (see e.g. here - anything below 9 maybe).
But basically all constants visible through py-lz4framed come straight from the lz4 library.
Edit: Update code example
See also help(lz4framed.compress) and https://github.com/Iotic-Labs/py-lz4framed/blob/master/lz4framed/__init__.py#L78
Thanks, http://python-lz4.readthedocs.io/en/latest/lz4.block.html exposes lz4.block.compress(source, mode='default', acceleration=1, compression=0) with default/fast/high_compression mode and acceleration/compression, so I guessed it could be fast with acceleration 8. But maybe they don't provide this in the latest version anymore?
Ah ok, I see now what you mean.
So the fast call options are only available through the block API. This project only supports the frame API (because this means it's interoperable whereas if you just compress raw blocks you need to decide e.g. how to transmit the length etc. yourself.) I guess you could use the python-lz4 module if you want to use the acceleration parameters (but as you can see here the frame API also doesn't expose acceleration). I can't comment on how stable that project is given I've never used it.
For the frame API the compression level is set only here (note no acceleration or other parameters). You could ask the lz4 project whether the frame API intends to support the accelerated parameter in the near future.
From what I can tell anything under lz4framed.LZ4F_COMPRESSION_MIN_HC (or here in C API) counts as "fast".
See also e.g. this gist from a while back where output side can be the same for low levels. (Obviously that test case is nonsense because the data is very synthetic.)
Thank you very much, that explains a lot and solves this issue! Thank you!
@dreamflasher - Once this is released (v1.7.6 of lz4 I think), accelerated mode will be possible. (No idea when lz4 plan to release though.)
For anyone else who'd like to use acceleration, new lz4 (with acceleration support) has no planned release for now. If you're interested in using it in lz4framed, post here to express interest.
Acceleration is now available (since v0.1.0)