/lizard

Lizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli (at low and medium compression levels) at decompression speed of 1000 MB/s and faster.

Primary LanguageCOtherNOASSERTION

Lizard - efficient compression with very fast decompression

Lizard (formerly LZ5) is a lossless compression algorithm which contains 4 compression methods:

  • fastLZ4 : compression levels -10...-19 are designed to give better decompression speed than LZ4 i.e. over 2000 MB/s
  • LIZv1 : compression levels -20...-29 are designed to give better ratio than LZ4 keeping 75% decompression speed
  • fastLZ4 + Huffman : compression levels -30...-39 add Huffman coding to fastLZ4
  • LIZv1 + Huffman : compression levels -40...-49 give the best ratio (comparable to zlib and low levels of zstd/brotli) at decompression speed of 1000 MB/s

Lizard library is based on frequently used LZ4 library by Yann Collet but the Lizard compression format is not compatible with LZ4. Lizard library is provided as open-source software using BSD 2-Clause license. The high compression/decompression speed is achieved without any SSE and AVX extensions.

Branch Status
lz5_v1.5 Build Status Build status
lizard Build Status Build status

Benchmarks

The following results are obtained with lzbench and -t16,16 using 1 core of Intel Core i5-4300U, Windows 10 64-bit (MinGW-w64 compilation under gcc 6.2.0) with silesia.tar which contains tarred files from Silesia compression corpus.

Compressor name Compression Decompress. Compr. size Ratio
memcpy 7332 MB/s 8719 MB/s 211947520 100.00
lz4 1.7.3 440 MB/s 2318 MB/s 100880800 47.60
lz4hc 1.7.3 -1 98 MB/s 2121 MB/s 87591763 41.33
lz4hc 1.7.3 -4 55 MB/s 2259 MB/s 79807909 37.65
lz4hc 1.7.3 -9 22 MB/s 2315 MB/s 77892285 36.75
lz4hc 1.7.3 -12 17 MB/s 2323 MB/s 77849762 36.73
lz4hc 1.7.3 -16 10 MB/s 2323 MB/s 77841782 36.73
lizard 1.0 -10 346 MB/s 2610 MB/s 103402971 48.79
lizard 1.0 -12 103 MB/s 2458 MB/s 86232422 40.69
lizard 1.0 -15 50 MB/s 2552 MB/s 81187330 38.31
lizard 1.0 -19 3.04 MB/s 2497 MB/s 77416400 36.53
lizard 1.0 -21 157 MB/s 1795 MB/s 89239174 42.10
lizard 1.0 -23 30 MB/s 1778 MB/s 81097176 38.26
lizard 1.0 -26 6.63 MB/s 1734 MB/s 74503695 35.15
lizard 1.0 -29 1.37 MB/s 1634 MB/s 68694227 32.41
lizard 1.0 -30 246 MB/s 909 MB/s 85727429 40.45
lizard 1.0 -32 94 MB/s 1244 MB/s 76929454 36.30
lizard 1.0 -35 47 MB/s 1435 MB/s 73850400 34.84
lizard 1.0 -39 2.94 MB/s 1502 MB/s 69807522 32.94
lizard 1.0 -41 126 MB/s 961 MB/s 76100661 35.91
lizard 1.0 -43 28 MB/s 1101 MB/s 70955653 33.48
lizard 1.0 -46 6.25 MB/s 1073 MB/s 65413061 30.86
lizard 1.0 -49 1.27 MB/s 1064 MB/s 60679215 28.63
zlib 1.2.8 -1 66 MB/s 244 MB/s 77259029 36.45
zlib 1.2.8 -6 20 MB/s 263 MB/s 68228431 32.19
zlib 1.2.8 -9 8.37 MB/s 266 MB/s 67644548 31.92
zstd 1.1.1 -1 235 MB/s 645 MB/s 73659468 34.75
zstd 1.1.1 -2 181 MB/s 600 MB/s 70168955 33.11
zstd 1.1.1 -5 88 MB/s 565 MB/s 65002208 30.67
zstd 1.1.1 -8 31 MB/s 619 MB/s 61026497 28.79
zstd 1.1.1 -11 16 MB/s 613 MB/s 59523167 28.08
zstd 1.1.1 -15 4.97 MB/s 639 MB/s 58007773 27.37
zstd 1.1.1 -18 2.87 MB/s 583 MB/s 55294241 26.09
zstd 1.1.1 -22 1.44 MB/s 505 MB/s 52731930 24.88
brotli 0.5.2 -0 217 MB/s 244 MB/s 78226979 36.91
brotli 0.5.2 -2 96 MB/s 283 MB/s 68066621 32.11
brotli 0.5.2 -5 24 MB/s 312 MB/s 60801716 28.69
brotli 0.5.2 -8 5.56 MB/s 324 MB/s 57382470 27.07
brotli 0.5.2 -11 0.39 MB/s 266 MB/s 51138054 24.13

Documentation

The raw Lizard block compression format is detailed within lizard_Block_format.

To compress an arbitrarily long file or data stream, multiple blocks are required. Organizing these blocks and providing a common header format to handle their content is the purpose of the Frame format, defined into lizard_Frame_format. Interoperable versions of Lizard must respect this frame format.