Handling too large inputs gracefully
Closed this issue · 6 comments
Currently, if the input is over 2 GB, the compression will fail.
python -c "import lzo; b=lzo.compress(str(bytearray((1024**3)*2)))"
Traceback (most recent call last):
File "", line 1, in
OverflowError: size does not fit in an int
Well that is definitely a significant issue. Thanks for such a succinct recreation of the issue.
OK, #46 advances this issue. This was done by making the entire library PY_SSIZE_T_CLEAN, which it probably should have been a long time ago, but I didn't know about until now.
Instead of the old limit of INT_MAX, we are now limited to LZO_UINT_MAX, which is the same as UINT_MAX. If the data exceeds that size, a better error message is given.
I'm going to call this closed for now.
Whoops, windows is busted, so closure was premature.
@jd-boyd I got this warning on Python 3.8.5
python -c "from lzo import *; print(LZO_VERSION_STRING.decode(), compress(b'').hex(' '), sep='\n')"
<string>:1: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
2.10
f0 00 00 00 00 11 00 00@keelung-yang Thanks for letting me know.