Failing tests: FATAL: Decompression failed and LZMA: Memory allocation error

Question

Failing tests: FATAL: Decompression failed and LZMA: Memory allocation error

szepeviktor opened this issue 10 years ago · 10 comments

make test with gcc version 4.9.1 (Debian 4.9.1-19)
./configoutput: https://gist.github.com/szepeviktor/6b26af8344bff1aa1284

Please help me.

Answer 1 · 2015-02-14T14:43:26.000Z

The VPS has 2GB of memory.
After stopping mysql etc. the LZMA memory error disappeared.
On low memory is it OK to throw allocation errors?

Answer 2 · 2015-02-14T14:43:44.000Z

FATALS

#################################################
# Simple compress and decompress
#################################################
Running ../../pcompress -c adapt -l 1 -s 1m /root/pcompress/test/datafiles/bin.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/bin.dat.pz /root/pcompress/test/datafiles/bin.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 1 -s 1m /root/pcompress/test/datafiles/share.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/share.dat.pz /root/pcompress/test/datafiles/share.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 1 -s 1m /root/pcompress/test/datafiles/inc.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/inc.dat.pz /root/pcompress/test/datafiles/inc.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 1 -s 1m /root/pcompress/test/datafiles/combined.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/combined.dat.pz /root/pcompress/test/datafiles/combined.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 1 -s 1m /root/pcompress/test/datafiles/comb_d.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/comb_d.dat.pz /root/pcompress/test/datafiles/comb_d.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 3 -s 1m /root/pcompress/test/datafiles/bin.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/bin.dat.pz /root/pcompress/test/datafiles/bin.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 3 -s 1m /root/pcompress/test/datafiles/share.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/share.dat.pz /root/pcompress/test/datafiles/share.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 3 -s 1m /root/pcompress/test/datafiles/inc.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/inc.dat.pz /root/pcompress/test/datafiles/inc.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 3 -s 1m /root/pcompress/test/datafiles/combined.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/combined.dat.pz /root/pcompress/test/datafiles/combined.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 3 -s 1m /root/pcompress/test/datafiles/comb_d.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/comb_d.dat.pz /root/pcompress/test/datafiles/comb_d.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 9 -s 1m /root/pcompress/test/datafiles/bin.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/bin.dat.pz /root/pcompress/test/datafiles/bin.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 9 -s 1m /root/pcompress/test/datafiles/share.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/share.dat.pz /root/pcompress/test/datafiles/share.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 9 -s 1m /root/pcompress/test/datafiles/inc.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/inc.dat.pz /root/pcompress/test/datafiles/inc.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 9 -s 1m /root/pcompress/test/datafiles/combined.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/combined.dat.pz /root/pcompress/test/datafiles/combined.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 9 -s 1m /root/pcompress/test/datafiles/comb_d.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/comb_d.dat.pz /root/pcompress/test/datafiles/comb_d.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/bin.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/bin.dat.pz /root/pcompress/test/datafiles/bin.dat.1
Scaling to 2 threads
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 100m /root/pcompress/test/datafiles/bin.dat
Scaling to 1 thread
Running ../../pcompress -d /root/pcompress/test/datafiles/bin.dat.pz /root/pcompress/test/datafiles/bin.dat.1
Scaling to 1 thread
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/share.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/share.dat.pz /root/pcompress/test/datafiles/share.dat.1
Scaling to 2 threads
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 100m /root/pcompress/test/datafiles/share.dat
Scaling to 1 thread
Running ../../pcompress -d /root/pcompress/test/datafiles/share.dat.pz /root/pcompress/test/datafiles/share.dat.1
Scaling to 1 thread
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/inc.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/inc.dat.pz /root/pcompress/test/datafiles/inc.dat.1
Scaling to 2 threads
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 100m /root/pcompress/test/datafiles/inc.dat
Scaling to 1 thread
Running ../../pcompress -d /root/pcompress/test/datafiles/inc.dat.pz /root/pcompress/test/datafiles/inc.dat.1
Scaling to 1 thread
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/combined.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/combined.dat.pz /root/pcompress/test/datafiles/combined.dat.1
Scaling to 2 threads
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 100m /root/pcompress/test/datafiles/combined.dat
Scaling to 1 thread
Running ../../pcompress -d /root/pcompress/test/datafiles/combined.dat.pz /root/pcompress/test/datafiles/combined.dat.1
Scaling to 1 thread
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/comb_d.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/comb_d.dat.pz /root/pcompress/test/datafiles/comb_d.dat.1
Scaling to 2 threads
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 100m /root/pcompress/test/datafiles/comb_d.dat
Scaling to 1 thread
Running ../../pcompress -d /root/pcompress/test/datafiles/comb_d.dat.pz /root/pcompress/test/datafiles/comb_d.dat.1
Scaling to 1 thread
Segmentation fault (core dumped)
FATAL: Decompression failed.

Answer 3 · 2015-02-14T14:52:27.000Z

Your precompiled pcompress_static_linux.tar.bz2 does not fail.

Answer 4 · 2015-02-14T14:54:43.000Z

Here is what I do:

# pbuilder login

cd /root
echo 'deb ftp://debian.mirrors.crysys.hu/debian/ jessie main contrib' > /etc/apt/sources.list
echo 'deb-src ftp://debian.mirrors.crysys.hu/debian/ jessie main contrib' >> /etc/apt/sources.list

# tools
apt-get update && apt-get install -y git ca-certificates mc \
    file libtool autoconf automake automake1.11 yasm

# deps
apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev libarchive-dev

# sources
git clone https://github.com/moinakg/pcompress.git
apt-get source libwavpack-dev

# build
cd pcompress/ && ./config --wavpack-dir=/root/wavpack-4.70.0; echo "config=$?"
make && make test

Answer 5 · 2015-02-14T15:16:46.000Z

The first fail is always the decompression of
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/bin.dat

Answer 6 · 2015-02-14T15:53:35.000Z

BTW. This is a very nice project: fast and very good on compression.

Answer 7 · 2015-02-14T16:57:47.000Z

Your precompiled pcompress_static_linux.tar.bz2 does not fail.

Not really:

# ./pcompress -c adapt -l 14 -s 1m ../pcompress/test/datafiles/bin.dat a.pz
Scaling to 2 threads
Minimum chunk size for Global Dedup must be 2097152 bytes

Error compressing file: /root/src/pcompress/test/datafiles/bin.dat

Answer 8 · 2015-02-15T02:42:19.000Z

Thanks a lot for reporting. This looks like a regression. I will check this
out.
I also have to improve detection of low memory systems and handle
appropriately.

Regards,
Moinak.

On Sat, Feb 14, 2015 at 8:57 AM, Viktor Szépe notifications@github.com
wrote:

Your precompiled pcompress_static_linux.tar.bz2 does not fail.

Not really:

./pcompress -c adapt -l 14 -s 1m ../pcompress/test/datafiles/bin.dat a.pz

Scaling to 2 threads
Minimum chunk size for Global Dedup must be 2097152 bytes

Error compressing file: /root/src/pcompress/test/datafiles/bin.dat

—
Reply to this email directly or view it on GitHub
#28 (comment).

http://moinakg.wordpress.com/

Answer 9 · 2015-04-18T11:50:11.000Z

@moinakg Is it the same issue?

# pcompress -a -j WordPress-4.1/ wp-pcompr
Scanning files.
Sorting ...
Scaling to 2 threads
Segmentation fault

2 GB free RAM. v3.1
Even with -t 1.

Answer 10 · 2015-04-29T08:55:23.000Z

Not sure. I have to reproduce this on a 2GB VM.