Avoid deadlock due to nested call in allocating large memory

[insuyun]

Hi, all.
If we use DieHarder for allocating a large memory, it will hang due to deadlock.
Here is poc.

#include <stdlib.h>
#include <stdio.h>

int main() {
  size_t xlsz = 1LL<<48;
  malloc(xlsz);
}

If I run this program in Linux 18.04, it will hang.
Here is my stacktrace.

#0  __lll_lock_wait (futex=futex@entry=0x7f83ef403fa8 <getCustomHeap()::buf+14792>, private=0) at lowlevellock.c:52
#1  0x00007f83ef0860a3 in __GI___pthread_mutex_lock (mutex=0x7f83ef403fa8 <getCustomHeap()::buf+14792>) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007f83ef389288 in HL::PosixLockType::lock (this=0x7f83ef403fa8 <getCustomHeap()::buf+14792>) at Heap-Layers/locks/posixlock.h:49
#3  std::lock_guard<HL::PosixLockType>::lock_guard (this=<optimized out>, __m=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_mutex.h:159
#4  HL::LockedHeap<HL::PosixLockType, CombineHeap<DieHardHeap<8, 7, 65536, false, false>, TheLargeHeap> >::malloc (this=0x7f83ef4005e0 <getCustomHeap()::buf>, sz=472) at Heap-Layers/heaps/threads/lockedheap.h:33
#5  HL::ANSIWrapper<HL::LockedHeap<HL::PosixLockType, CombineHeap<DieHardHeap<8, 7, 65536, false, false>, TheLargeHeap> > >::malloc (this=0x7f83ef4005e0 <getCustomHeap()::buf>, sz=472)
    at Heap-Layers/wrappers/ansiwrapper.h:57
#6  xxmalloc (sz=<optimized out>) at source/libdieharder.cpp:83
#7  0x00007f83ef1282f3 in _IO_new_fdopen (fd=fd@entry=420, mode=<optimized out>, mode@entry=0x7f83ef25a64c "w+") at iofdopen.c:122
#8  0x00007f83ef10860c in __GI_perror (s=0x7f83ef3bd17b "MmapWrapper") at perror.c:67
#9  0x00007f83ef3bc62f in HL::MmapWrapper::map (sz=<optimized out>) at Heap-Layers/wrappers/mmapwrapper.h:163
#10 LargeHeap<HL::MmapWrapper>::malloc (this=0x7f83ef404260 <HL::singleton<LargeHeap<HL::MmapWrapper> >::getInstance()::buf>, sz=281474976710656) at include/largeheap.h:17
#11 0x00007f83ef3892a1 in HL::OneHeap<LargeHeap<HL::MmapWrapper> >::malloc (sz=281474976710656) at Heap-Layers/heaps/./utility/oneheap.h:17
#12 TheLargeHeap::malloc (this=<optimized out>, sz=281474976710656) at source/libdieharder.cpp:46
#13 CombineHeap<DieHardHeap<8, 7, 65536, false, false>, TheLargeHeap>::malloc (this=0x7f83ef4005e0 <getCustomHeap()::buf>, sz=281474976710656) at include/layers/combineheap.h:22
#14 HL::LockedHeap<HL::PosixLockType, CombineHeap<DieHardHeap<8, 7, 65536, false, false>, TheLargeHeap> >::malloc (this=0x7f83ef4005e0 <getCustomHeap()::buf>, sz=281474976710656)
    at Heap-Layers/heaps/threads/lockedheap.h:34
#15 HL::ANSIWrapper<HL::LockedHeap<HL::PosixLockType, CombineHeap<DieHardHeap<8, 7, 65536, false, false>, TheLargeHeap> > >::malloc (this=0x7f83ef4005e0 <getCustomHeap()::buf>, sz=281474976710656)
    at Heap-Layers/wrappers/ansiwrapper.h:57
#16 xxmalloc (sz=<optimized out>) at source/libdieharder.cpp:83
#17 0x000055f6f590316f in main ()

As you can see from my stack trace, the lock is acquired twice because perror also uses DieHarder for its its fdopen().
We can avoid this by using recursive mutex or by removing this perror (which is only for an error message).

Best,
Insu Yun.