malloc() memory corruption error
willyspinner opened this issue · 14 comments
Hi reaper,
I was running parallel reaper processes as worker processes spawned by python's multiprocessing Pool to speed up generation of .f0 files from multiple single-channel .wav files. In each worker process, I call
os.system("reaper -i FILENAME.wav -f FILENAME.f0 -e 0.02 -a")which executes the reaper command processing that filename in a subshell (Obviously I made sure that no two reaper workers access the same FILENAME).
All of a sudden, I got an unexpected malloc error below.
(FYI I am running Ubuntu 16.04 LTS, but have confirmed this error persists on Mac OSX as well...)
Residual symmetry: P:837.185303 N:803.405762 MEAN:-0.159036
Inverting signal
*** Error in `reaper': malloc(): memory corruption: 0x0000000006212a70 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ff4097747e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7ff40977f13e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7ff409781184]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znwm+0x18)[0x7ff40a073e78]
reaper(_Z12MakeF0OutputR12EpochTrackerfPP5Track+0x82)[0x415fcf]
reaper(_Z18ComputeEpochsAndF0R12EpochTrackerffPP5TrackS3_S3_+0x119)[0x416361]
reaper(main+0x4c1)[0x416879]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ff40971d830]
reaper(_start+0x29)[0x415ce9]
======= Memory map: ========
00400000-0043c000 r-xp 00000000 08:01 794191 /usr/bin/reaper
0063b000-0063c000 r--p 0003b000 08:01 794191 /usr/bin/reaper
0063c000-0063d000 rw-p 0003c000 08:01 794191 /usr/bin/reaper
01686000-06230000 rw-p 00000000 00:00 0 [heap]
7ff404000000-7ff404021000 rw-p 00000000 00:00 0
7ff404021000-7ff408000000 ---p 00000000 00:00 0
7ff40945b000-7ff40961c000 rw-p 00000000 00:00 0
7ff4096fd000-7ff4098bd000 r-xp 00000000 08:01 394095 /lib/x86_64-linux-gnu/libc-2.23.so
7ff4098bd000-7ff409abd000 ---p 001c0000 08:01 394095 /lib/x86_64-linux-gnu/libc-2.23.so
7ff409abd000-7ff409ac1000 r--p 001c0000 08:01 394095 /lib/x86_64-linux-gnu/libc-2.23.so
7ff409ac1000-7ff409ac3000 rw-p 001c4000 08:01 394095 /lib/x86_64-linux-gnu/libc-2.23.so
7ff409ac3000-7ff409ac7000 rw-p 00000000 00:00 0
7ff409ac7000-7ff409add000 r-xp 00000000 08:01 394116 /lib/x86_64-linux-gnu/libgcc_s.so.1
7ff409add000-7ff409cdc000 ---p 00016000 08:01 394116 /lib/x86_64-linux-gnu/libgcc_s.so.1
7ff409cdc000-7ff409cdd000 rw-p 00015000 08:01 394116 /lib/x86_64-linux-gnu/libgcc_s.so.1
7ff409cdd000-7ff409de5000 r-xp 00000000 08:01 394127 /lib/x86_64-linux-gnu/libm-2.23.so
7ff409de5000-7ff409fe4000 ---p 00108000 08:01 394127 /lib/x86_64-linux-gnu/libm-2.23.so
7ff409fe4000-7ff409fe5000 r--p 00107000 08:01 394127 /lib/x86_64-linux-gnu/libm-2.23.so
7ff409fe5000-7ff409fe6000 rw-p 00108000 08:01 394127 /lib/x86_64-linux-gnu/libm-2.23.so
7ff409fe6000-7ff40a158000 r-xp 00000000 08:01 394983 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7ff40a158000-7ff40a358000 ---p 00172000 08:01 394983 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7ff40a358000-7ff40a362000 r--p 00172000 08:01 394983 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7ff40a362000-7ff40a364000 rw-p 0017c000 08:01 394983 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7ff40a364000-7ff40a368000 rw-p 00000000 00:00 0
7ff40a368000-7ff40a38e000 r-xp 00000000 08:01 394075 /lib/x86_64-linux-gnu/ld-2.23.so
7ff40a3e6000-7ff40a4b1000 rw-p 00000000 00:00 0
7ff40a517000-7ff40a582000 rw-p 00000000 00:00 0
7ff40a58a000-7ff40a58d000 rw-p 00000000 00:00 0
7ff40a58d000-7ff40a58e000 r--p 00025000 08:01 394075 /lib/x86_64-linux-gnu/ld-2.23.so
7ff40a58e000-7ff40a58f000 rw-p 00026000 08:01 394075 /lib/x86_64-linux-gnu/ld-2.23.so
7ff40a58f000-7ff40a590000 rw-p 00000000 00:00 0
7ffd8caad000-7ffd8cace000 rw-p 00000000 00:00 0 [stack]
7ffd8cbee000-7ffd8cbf0000 r--p 00000000 00:00 0 [vvar]
7ffd8cbf0000-7ffd8cbf2000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted
Could there be something I'm doing wrong? Or could Reaper be thread-unsafe? I just recently started using REAPER, and would really like to use this simple and powerful command-line tool frequently in the future.
Thank you for reading my issue :)
Best,
Wilson
Those are parallel processes so it has nothing to do with reaper being thread safe. What you describe is a memory corruption problem, maybe some memory is not freed correctly.
Could you share the .wav file that is causing the error?
In addition you could take that .wav file that causes an error, compile reaper in debug mode and see if there's any hint in the output logging information.
Thanks
Hello,
so I looked into the wav files that caused the memory corruption errors, and in total, there were only 50 files that couldn't be processed out of 1000 files. I realised that some of the files were in fact, stereo, and I realised that reaper expects mono input. So for those, I will simply have to preprocess the wav files into mono first.
However, some mono ones still caused the error above. Here are some samples of the wav files that caused the errors (obtained by MAC OS's afinfo command):
File type ID: WAVE
Num Tracks: 1
----
Data format: 1 ch, 48000 Hz, 'lpcm' (0x0000000C) 16-bit little-endian signed integer
no channel layout.
estimated duration: 10.860000 sec
audio bytes: 1042560
audio packets: 521280
bit rate: 768000 bits per second
packet size upper bound: 2
maximum packet size: 2
audio data file offset: 78
optimized
source bit depth: I16
----
here is another one:
File type ID: WAVE
Num Tracks: 1
----
Data format: 1 ch, 48000 Hz, 'lpcm' (0x0000000C) 16-bit little-endian signed integer
no channel layout.
estimated duration: 20.400000 sec
audio bytes: 1958400
audio packets: 979200
bit rate: 768000 bits per second
packet size upper bound: 2
maximum packet size: 2
audio data file offset: 78
optimized
source bit depth: I16
----
Whereas the ones successfully processed by reaper often have this format:
File type ID: WAVE
Num Tracks: 1
----
Data format: 1 ch, 48000 Hz, 'lpcm' (0x0000000C) 16-bit little-endian signed integer
no channel layout.
estimated duration: 19.020000 sec
audio bytes: 1825920
audio packets: 912960
bit rate: 768000 bits per second
packet size upper bound: 2
maximum packet size: 2
audio data file offset: 78
optimized
source bit depth: I16
----
Could there be an audio format issue that makes Reaper corrupt memory?
If you'd like, I could share you some of the actual samples of the wav files.
Thank you so much for your help,
Wilson
I'm working with @willyspinner on this project. I will try to provide more information later including some failing samples.
In running this across our samples now, I see Reaper exit code -11 for several failing samples, though when that exit code is returned there is no stdout or stderr from Reaper.
I see the memory corruption accompanied by exit code -6.
Will get back soon with more info.
Some of the failing samples are empty or sound empty (maybe corrupt?). @xavigonzalvo is there a way I can send you the samples privately? I don't want to post them in here as they are somewhat sensitive in nature.
By debug mode, did you mean like cmake -DCMAKE_BUILD_TYPE=Debug ..?
Just wanted to let you know I managed to "fix" the samples that still caused crashes despite being playable in an ordinary media player. I just ran ffmpeg -i <file> -ac 1 <output_file> which is technically a no-op for these since they were already mono (1 channel), but it seemed to fix the problem. Notably in the ffmpeg output for those operations, there was a yellow message reading Guessed Channel Layout for Input Stream #0.0 : mono.
I would consider this closed for my practical purposes but if you want to keep debugging, LMK and I can get you some samples that surface the problem.
I believe so... ffmpeg was able to repair the files somehow apparently.
Dear sir,
I also met the same problem, and I cannot deal with it.
when I used the option -e, and it was set to default value 0.005, the code is ok.
reaper -i 40fa0110_1.1436_028c0216_-1.1436_40fo030o.wav -f ./tmp/bla.f0 -p ./tmp/bla.pm -e 0.005 -a
But, when I set the -e parameter to 0.016, the code had a problem: malloc() memory corruption error. The details are:
So, could you help me to deal with the problem?
The wav I used is upload on: https://pan.baidu.com/s/1CqL-q761FLJeSuelLoTHow
and the link password is: zesd
@gemengtju Another thing I've done with some success is convert to 44100 sample rate—i.e. ffmpeg arg -ar 44100—but even that does not always work. It fixes some files but still some remain unable to process.
Is your wav file of a reasonable length and actually playable in other media players?
can you drop the wav somewhere accessible ? (there are some pop-ups I don't understand in the link above)
I have met this problem when I use a large resample_interval in ResampleAndReturnResults(float resample_interval, ...), it happened on some specific wave files:
for wav in wav_list:
CHECK(et->Init(wav));
CHECK(et->ComputeFeatures());
CHECK(et->TrackEpochs());
// extract f0 & corr
std::vector<float> f0;
std::vector<float> corr;
CHECK(et->ResampleAndReturnResults(resample_interval, &f0, &corr));
But when I test on the single file, no error occurs.
Fixed by using LD_PRELOAD=/usr/lib/libtcmalloc.so python whatever.py
Edit: this is actually for the Python PyREAPER wrapper