SPRING is a compression tool for Fastq files (containing up to 4.29 Billion reads):
- Near-optimal compression ratios for single-end and paired-end datasets
- Fast and memory-efficient decompression
- Supports variable length short reads of length upto 511 bases (without -l flag)
- Supports variable length long reads of arbitrary length (upto 4.29 Billion) (with -l flag). This mode directly applies general purpose compression (BSC) to reads and so compression gains might be lower than those without -l flag.
- Supports lossless compression of reads, quality scores and read identifiers
- Supports reordering of reads (while preserving read pairing information) to boost compression
- Supports quantization of quality values using QVZ, Illumina 8-level binning and binary thresholding
- Supports decompression of a subset of reads (random access)
- Supports gzipped fastq files as input (output) during (de)compression
- Tested on Linux and macOS
git clone https://github.com/shubhamchandak94/SPRING.gitThe instructions below will create the spring executable in the build directory inside SPRING. If you plan to build and run SPRING on separate architectures, then you might need to remove/comment the line set(FLAGS "${FLAGS} -march=native") in CMakeLists.txt (or use flags based on the target architecture).
On Linux with cmake installed and version at least 3.9 (check using cmake --version):
cd SPRING
mkdir build
cd build
cmake ..
makeOn Linux with cmake not installed or with version older than 3.9:
cd SPRING
mkdir build
cd build
wget https://cmake.org/files/v3.10/cmake-3.10.3.tar.gz
tar -xzf cmake-3.10.3.tar.gz
cd cmake-3.10.3
./configure
make
cd ..
./cmake-3.10.3/bin/cmake ..
makeOn macOS, install GCC compiler since Clang has issues with OpenMP library:
- Install HomeBrew (https://brew.sh/)
- Install GCC (this step will be faster if Xcode command line tools are already installed using
xcode-select --install):
brew update
brew install gcc@7- Set environment variables:
export CC=gcc-7
export CXX=g++-7- Delete
CMakeCache.txt(if present) from the build directory - Follow the steps above for Linux
Run the spring executable /PATH/TO/spring with the options below:
Allowed options:
-h [ --help ] produce help message
-c [ --compress ] compress
-d [ --decompress ] decompress
--decompress-range arg --decompress-range start end
(optional) decompress only reads (or read
pairs for PE datasets) from start to end
(both inclusive) (1 <= start <= end <=
num_reads (or num_read_pairs for PE)). If -r
was specified during compression, the range
of reads does not correspond to the original
order of reads in the FASTQ file.
-i [ --input-file ] arg input file name (two files for paired end)
-o [ --output-file ] arg output file name (for paired end
decompression, if only one file is specified,
two output files will be created by suffixing
.1 and .2.)
-w [ --working-dir ] arg (=.) directory to create temporary files (default
current directory)
-t [ --num-threads ] arg (=8) number of threads (default 8)
-r [ --allow-read-reordering ] do not retain read order during compression
(paired reads still remain paired)
--no-quality do not retain quality values during
compression
--no-ids do not retain read identifiers during
compression
-q [ --quality-opts ] arg quality mode: possible modes are
1. -q lossless (default)
2. -q qvz qv_ratio (QVZ lossy compression,
parameter qv_ratio roughly corresponds to
bits used per quality value)
3. -q ill_bin (Illumina 8-level binning)
4. -q binary thr high low (binary (2-level)
thresholding, quality binned to high if >=
thr and to low if < thr)
-l [ --long ] Use for compression of arbitrarily long read
lengths. Can also provide better compression
for reads with significant number of indels.
-r disabled in this mode. For Illumina short
reads, compression is better without -l flag.
-g [ --gzipped_fastq ] enable if compression input is gzipped fastq
or to output gzipped fastq during
decompression
Note that the SPRING compressed files are tar archives consisting of the different compressed streams, although we recommend using the .spring extension as in the examples shown below.
This section contains several examples for SPRING compression and decompression with various modes and options. The compressed SPRING file uses the .spring extension as a convention.
For compressing file_1.fastq and file_2.fastq losslessly using default 8 threads (Lossless).
./spring -c -i file_1.fastq file_2.fastq -o file.springFor compressing file_1.fastq.gz and file_2.fastq.gz (gzipped fastq files) losslessly using default 8 threads (Lossless).
./spring -c -i file_1.fastq.gz file_2.fastq.gz -o file.spring -gUsing 16 threads (Lossless).
./spring -c -i file_1.fastq file_2.fastq -o file.spring -t 16Compressing with only paired end info preserved, ids not stored, qualities compressed after Illumina binning (Recommended lossy mode for older Illumina machines. For Novaseq files, lossless quality compression is recommmended).
./spring -c -i file_1.fastq file_2.fastq -r --no-ids -q ill_bin -o file.springCompressing with only paired end info preserved, ids not stored, qualities binary thresholded (qv < 20 binned to 6 and qv >= 20 binned to 40).
./spring -c -i file_1.fastq file_2.fastq -r --no-ids -q binary 20 40 6 -o file.springCompressing with only paired end info preserved, ids not stored, qualities quantized using qvz with approximately 1 bit used per quality value.
./spring -c -i file_1.fastq file_2.fastq -r --no-ids -q qvz 1.0 -o file.springCompressing only reads and ids.
./spring -c -i file_1.fastq file_2.fastq --no-quality -o file.springCompressing single-end long read Fastq losslessly.
./spring -c -l -i file.fastq -o file.springFor single end file, compressing without order preserved.
./spring -c -i file.fastq -r -o file.springFor single end file, compressing with order preserved (lossless).
./spring -c -i file.fastq -o file.springDecompressing (single end) to file.fastq.
./spring -d -i file.spring -o file.fastqDecompressing (single end) to file.fastq, only decompress reads from 400 to 10000000.
./spring -d -i file.spring -o file.fastq --decompress-range 400 1000000Decompressing (paired end) to file.fastq.1 and file.fastq.2.
./spring -d -i file.spring -o file.fastqDecompressing (paired end) to file_1.fastq and file_2.fastq.
./spring -d -i file.spring -o file_1.fastq file_2.fastqDecompressing (paired end) to file_1.fastq.gz and file_2.fastq.gz.
./spring -d -i file.spring -o file_1.fastq.gz file_2.fastq.gz -gDecompressing (paired end) to file_1.fastq and file_2.fastq, only decompress pairs from 4000000 to 8000000.
./spring -d -i file.spring -o file_1.fastq file_2.fastq --decompress-range 4000000 8000000