MacHu-GWU/compress-project

All in one data compression library.

---

Welcome to compress Documentation

compress provides a unified interface for various mature data compression algorithms. It supports algorithms from both the Python Standard Library and the community, offering a range of options for different compression needs.

Supported Algorithms

From Python Standard library

• zlib.
• bz2.
• lzma, high compression ratio but slow.

From Community (Additional Library Required)

• snappy, from Google, lower compression ratio but super fast! (on MacOS, you need to install it via brew install snappy, on Ubuntu, you need sudo apt-get install libsnappy-dev.
• lz4, lower ratio, super fast!
• pyzstd, very fast!

Note: Community libraries are not installed by default with compress. To include them, use:

pip install compress[lz4,snappy,zstd]

These libraries require a C compiler. If you encounter issues installing the C compiler for your OS, refer to this tutorial.

Usage Example

import sys
import compress.api as compress

data = ("hello world" * 1000).encode("utf-8")
print(f"before: {sys.getsizeof(data)}")

data_compressed = compress.compress(
    algo=compress.Algorithm.gzip,
    data=data,
    kwargs={"compresslevel": 9},
)
print(f"after: {sys.getsizeof(data_compressed)}")

Benchmark

This website provides comprehensive comparison and visualization. But how do you know how it works on your own production environment?.

compress comes with a tool to run benchmark test for All test case, All algorithm, All parameters, and you will get informative stats about ratio, compress/decompress speed in ascii table format. Then You are able to visualize it in the way you preferred.

To run benchmark test, just do:

pip install -r requirements-benchmark.txt
python ./benchmark/run_benchmark.py

Then you can find the result at benchmark/result.txt.

Install

compress is released on PyPI, so all you need is to:

$ pip install compress

To upgrade to latest version:

$ pip install --upgrade compress