codership/glb

Galera Load Balancer connections crush

smartdoc opened this issue · 5 comments

smartdoc commented

Hi there!

I try to connect more than 2 clients to galera cluster through glb.
What have i got:

root@galera1:~/glb-1.0.1# glbd -w exec:"/root/glb-1.0.1/files/mysql.sh -uroot -proot" -m 512 -t 3 3306 galera3:3308 galera2:3308 galera1:3308

Watchdog:

Address : exp setw state lat curw
192.168.55.115:3308 : + 1.000 READY 0.18607 1.000
192.168.55.114:3308 : + 1.000 READY 0.17800 1.000
127.0.1.1:3308 : + 1.000 READY 0.17787 1.000
Destinations: 3

Router:
Address : weight usage map conns
127.0.1.1:3308 : 1.000 0.500 N/A 1
192.168.55.114:3308 : 1.000 0.500 N/A 1
192.168.55.115:3308 : 1.000 0.500 N/A 1
Destinations: 3, total connections: 3 of 512 max

Pool: connections per thread: 1 1 1

glbd: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
INFO: glb_signal.c:42: Received signal 6. Terminating.
Aborted

I can send you virtualbox vm with galera cluster and glb.
Please, fix it.
Sincerely,
from Russia, with love.

ayurchen commented

Sorry for late reply. As you can see, it looks very much like a system issue, at least it reports assertion failure in malloc code. This of course is very unlikely, but to find the offending code in glbd we need a backtrace.

For that, please

  1. build glbd with debug option: --with-debug
  2. enable core dumps in the shell session:
    $ ulimit -c unlimited
  3. start glbd in that session
  4. when it aborts, locate the core file (usually 'core' in the working directory)
  5. run
    $ gdb -c -batch -ex "thr apply all bt" > stack.dump

Post stack.dump here, together with the glbd command line.

Thanks,
Alex

ayurchen commented

Don't remove the core file, it may be needed for further investigation.

smartdoc commented

Hi, Alex.
Regarding to this problem.
i reconfigure glb, and install it.
Then, try to start it:
root@galera1:~/glb-1.0.1# glbd -w exec:"/root/glb-1.0.1/files/mysql.sh -uroot -proot" -t 3 0.0.0.0:3306 galera1:3308 galera2:3308 galera3:3308
glb v1.0.1 (epoll)
Incoming address: 0.0.0.0:3306, control FIFO: /tmp/glbd.fifo
Control address: none
Number of threads: 3, max conn: 493, nodelay: ON, keepalive: ON, defer accept: OFF, linger: OFF, daemon: NO, lat.count: 0, policy: 'least connected', top: NO, verbose: NO
Destinations: 3
0: 127.0.1.1:3308 , w: 1.000
1: 192.168.55.114:3308 , w: 1.000
2: 192.168.55.115:3308 , w: 1.000
Watchdog:
Address : exp setw state lat curw
127.0.1.1:3308 : + 1.000 READY 0.21660 1.000
192.168.55.114:3308 : + 1.000 NOTFOUND 0.00000 -1.000
192.168.55.115:3308 : + 1.000 NOTFOUND 0.00000 -1.000
Destinations: 3

Router:
Address : weight usage map conns
127.0.1.1:3308 : 1.000 0.000 N/A 0
Destinations: 1, total connections: 0 of 493 max

Pool: connections per thread: 0 0 0

*** glibc detected *** glbd: realloc(): invalid pointer: 0x095961c0 ***
======= Backtrace: =========
/lib/i386-linux-gnu/i686/cmov/libc.so.6(+0x70f01)[0xb7673f01]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(realloc+0x2bb)[0xb7679acb]
glbd[0x804d15f]
======= Memory map: ========
08048000-0805c000 r-xp 00000000 08:01 13804 /usr/sbin/glbd
0805c000-0805d000 rw-p 00013000 08:01 13804 /usr/sbin/glbd
09588000-095a9000 rw-p 00000000 00:00 0 [heap]
b2d08000-b2d24000 r-xp 00000000 08:01 129301 /lib/i386-linux-gnu/libgcc_s.so.1
b2d24000-b2d25000 rw-p 0001b000 08:01 129301 /lib/i386-linux-gnu/libgcc_s.so.1
b2d2a000-b2d2b000 ---p 00000000 00:00 0
b2d2b000-b352b000 rw-p 00000000 00:00 0
b352b000-b352c000 ---p 00000000 00:00 0
b352c000-b3d2c000 rw-p 00000000 00:00 0
b3d2c000-b3d2d000 ---p 00000000 00:00 0
b3d2d000-b452f000 rw-p 00000000 00:00 0
b452f000-b4530000 ---p 00000000 00:00 0
b4530000-b4d30000 rw-p 00000000 00:00 0
b4d30000-b4d31000 ---p 00000000 00:00 0
b4d31000-b5531000 rw-p 00000000 00:00 0
b5531000-b5532000 ---p 00000000 00:00 0
b5532000-b5d32000 rw-p 00000000 00:00 0
b5d32000-b5d33000 ---p 00000000 00:00 0
b5d33000-b6533000 rw-p 00000000 00:00 0
b6533000-b6534000 ---p 00000000 00:00 0
b6534000-b6d34000 rw-p 00000000 00:00 0
b6d34000-b6d35000 ---p 00000000 00:00 0
b6d35000-b75f6000 rw-p 00000000 00:00 0
b75f6000-b7600000 r-xp 00000000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b7600000-b7601000 r--p 00009000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b7601000-b7602000 rw-p 0000a000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b7602000-b7603000 rw-p 00000000 00:00 0
b7603000-b775f000 r-xp 00000000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b775f000-b7760000 ---p 0015c000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b7760000-b7762000 r--p 0015c000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b7762000-b7763000 rw-p 0015e000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b7763000-b7767000 rw-p 00000000 00:00 0
b7767000-b777c000 r-xp 00000000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b777c000-b777d000 r--p 00014000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b777d000-b777e000 rw-p 00015000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b777e000-b7780000 rw-p 00000000 00:00 0
b7780000-b7782000 r-xp 00000000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b7782000-b7783000 r--p 00001000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b7783000-b7784000 rw-p 00002000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b7784000-b778b000 rw-p 00000000 00:00 0
b778b000-b778c000 r-xp 00000000 00:00 0 [vdso]
b778c000-b77a8000 r-xp 00000000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
b77a8000-b77a9000 r--p 0001b000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
b77a9000-b77aa000 rw-p 0001c000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
bfa15000-bfa36000 rw-p 00000000 00:00 0 [stack]
INFO: glb_signal.c:42: Received signal 6. Terminating.
Aborted (core dumped)

It's created core dump file.
I make analyze it and get the next thing:

root@galera1:/glb-1.0.1# gdb -c core -batch -ex "thr apply all bt" > stack.dump
root@galera1:/glb-1.0.1# more stack.dump
[New LWP 18833]
[New LWP 18874]
[New LWP 18875]
[New LWP 18818]
[New LWP 18819]
[New LWP 18820]
[New LWP 18821]
[New LWP 18822]
[New LWP 18824]
[New LWP 18828]
Core was generated by `glbd -w exec /root/glb-1.0.1/files/mysql.sh -uroot -proot -t 3 0.0.
0.0:3306 gal'.
Program terminated with signal 6, Aborted.
#0 0xb778b422 in __kernel_vsyscall ()

Thread 10 (LWP 18828):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb7771703 in ?? ()
#2 0x00000000 in ?? ()

Thread 9 (LWP 18824):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb7771703 in ?? ()
#2 0x00000000 in ?? ()

Thread 8 (LWP 18822):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb7771703 in ?? ()
#2 0x00000000 in ?? ()

Thread 7 (LWP 18821):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76d91f6 in ?? ()
#2 0x00000000 in ?? ()

Thread 6 (LWP 18820):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76d91f6 in ?? ()
#2 0x00000000 in ?? ()

Thread 5 (LWP 18819):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76d91f6 in ?? ()
#2 0x00000000 in ?? ()

Thread 4 (LWP 18818):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76a3326 in ?? ()
#2 0x00000000 in ?? ()

Thread 3 (LWP 18875):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76da3f8 in ?? ()
#2 0xb352a334 in ?? ()
#3 0xb76d878e in ?? ()

Thread 2 (LWP 18874):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76cad96 in ?? ()
#2 0x000003e8 in ?? ()
#3 0x00000001 in ?? ()
#4 0x09596200 in ?? ()
#5 0x0804ee40 in ?? ()
#6 0x09596224 in ?? ()
#7 0xb76d878e in ?? ()

Thread 1 (LWP 18833):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb762d941 in ?? ()
#2 0xb7761ff4 in ?? ()
#3 0xb7630d02 in ?? ()
#4 0x00000006 in ?? ()
#5 0xb452b924 in ?? ()
#6 0xb7669e15 in ?? ()
#7 0x00000016 in ?? ()
#8 0xb452bb70 in ?? ()
#9 0xb7673f01 in ?? ()
#10 0x00000002 in ?? ()
#11 0xb7741e70 in ?? ()
#12 0xbfa35e22 in ?? ()
#13 0xb7679acb in ?? ()
#14 0x095961c0 in ?? ()
#15 0x0804d15f in ?? ()
#16 0x095961c0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

That's all. Don't hesitate to push me if you need something else ;)
I wish you a Merry Christmas and happy holidays :)

Regards,
V.

ayurchen commented

Hi!

  1. You forgot to include path to glbd binary in tha arguments to gdb. As a result it failed to find symbols definitions.
  2. installing glbd may lead to stripping the debug info from it. Don't install. Run glbd from where it was built. E.g.

root@galera1:/glb-1.0.1# ./src/glbd -w exec:"/root/glb-1.0.1/files/mysql.sh -uroot -proot" -t 3 0.0.0.0:3306 galera1:3308 galera2:3308 galera3:3308
root@galera1:/glb-1.0.1# gdb ./src/glbd -c core -batch -ex "thr apply all bt" > stack.dump

  1. You don't have to be root for that. And you better not be root.

Regards,
Alex

smartdoc commented

Sorry for late response. I was on vacation and throw out computer :)
Well, about glb.
3rd point doesn't matter, cause of i use virtual machines.
What have i got using your recommendations:

root@galera1:/glb-1.0.1# ulimit -c unlimited
root@galera1:/glb-1.0.1# ./src/glbd -w exec:"/root/glb-1.0.1/files/mysql.sh -uroot -proot" -t 3 0.0.0.0:3306 galera1:3308 galera2:3308 galera3:3308
glb v1.0.1 (epoll)
Incoming address: 0.0.0.0:3306, control FIFO: /tmp/glbd.fifo
Control address: none
Number of threads: 3, max conn: 493, nodelay: ON, keepalive: ON, defer accept: OFF, linger: OFF, daemon: NO, lat.count: 0, policy: 'least connected', top: NO, verbose: NO
Destinations: 3
0: 127.0.1.1:3308 , w: 1.000
1: 192.168.55.114:3308 , w: 1.000
2: 192.168.55.115:3308 , w: 1.000
Watchdog:

    Address       : exp  setw     state    lat     curw
  127.0.1.1:3308  :  +   1.000    READY  0.20919   1.000

192.168.55.114:3308 : + 1.000 READY 0.21450 1.000
192.168.55.115:3308 : + 1.000 NOTFOUND 0.00000 -1.000

Destinations: 3

Router:

    Address       :   weight   usage    map  conns

192.168.55.114:3308 : 1.000 0.000 N/A 0
127.0.1.1:3308 : 1.000 0.000 N/A 0

Destinations: 2, total connections: 0 of 493 max

Pool: connections per thread: 0 0 0

*** glibc detected *** ./src/glbd: realloc(): invalid pointer: 0x09481208 ***
===== Backtrace: =====
/lib/i386-linux-gnu/i686/cmov/libc.so.6(+0x70f01)[0xb761ff01]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(realloc+0x2bb)[0xb7625acb]
./src/glbd[0x804cd75]
./src/glbd[0x805051e]
./src/glbd[0x80507d7]
/lib/i386-linux-gnu/i686/cmov/libpthread.so.0(+0x5c39)[0xb7718c39]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(clone+0x5e)[0xb768478e]
===== Memory map: =====
08048000-0805d000 r-xp 00000000 08:01 135397 /root/glb-1.0.1/src/glbd
0805d000-0805e000 rw-p 00014000 08:01 135397 /root/glb-1.0.1/src/glbd
09473000-09494000 rw-p 00000000 00:00 0 [heap]
b2cb4000-b2cd0000 r-xp 00000000 08:01 129301 /lib/i386-linux-gnu/libgcc_s.so.1
b2cd0000-b2cd1000 rw-p 0001b000 08:01 129301 /lib/i386-linux-gnu/libgcc_s.so.1
b2cd6000-b2cd7000 ---p 00000000 00:00 0
b2cd7000-b34d7000 rw-p 00000000 00:00 0
b34d7000-b34d8000 ---p 00000000 00:00 0
b34d8000-b3cd8000 rw-p 00000000 00:00 0
b3cd8000-b3cd9000 ---p 00000000 00:00 0
b3cd9000-b44db000 rw-p 00000000 00:00 0
b44db000-b44dc000 ---p 00000000 00:00 0
b44dc000-b4cdc000 rw-p 00000000 00:00 0
b4cdc000-b4cdd000 ---p 00000000 00:00 0
b4cdd000-b54dd000 rw-p 00000000 00:00 0
b54dd000-b54de000 ---p 00000000 00:00 0
b54de000-b5cde000 rw-p 00000000 00:00 0
b5cde000-b5cdf000 ---p 00000000 00:00 0
b5cdf000-b64df000 rw-p 00000000 00:00 0
b64df000-b64e0000 ---p 00000000 00:00 0
b64e0000-b6ce0000 rw-p 00000000 00:00 0
b6ce0000-b6ce1000 ---p 00000000 00:00 0
b6ce1000-b75a2000 rw-p 00000000 00:00 0
b75a2000-b75ac000 r-xp 00000000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b75ac000-b75ad000 r--p 00009000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b75ad000-b75ae000 rw-p 0000a000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b75ae000-b75af000 rw-p 00000000 00:00 0
b75af000-b770b000 r-xp 00000000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b770b000-b770c000 ---p 0015c000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b770c000-b770e000 r--p 0015c000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b770e000-b770f000 rw-p 0015e000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b770f000-b7713000 rw-p 00000000 00:00 0
b7713000-b7728000 r-xp 00000000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b7728000-b7729000 r--p 00014000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b7729000-b772a000 rw-p 00015000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b772a000-b772c000 rw-p 00000000 00:00 0
b772c000-b772e000 r-xp 00000000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b772e000-b772f000 r--p 00001000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b772f000-b7730000 rw-p 00002000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b7730000-b7737000 rw-p 00000000 00:00 0
b7737000-b7738000 r-xp 00000000 00:00 0 [vdso]
b7738000-b7754000 r-xp 00000000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
b7754000-b7755000 r--p 0001b000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
b7755000-b7756000 rw-p 0001c000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
bfc54000-bfc75000 rw-p 00000000 00:00 0 [stack]
INFO: glb_signal.c:42: Received signal 6. Terminating.
Aborted (core dumped)

After it, dumped core was delivered to gdb.

root@galera1:~/glb-1.0.1# gdb ./src/glbd -c core -batch -ex "thr apply all bt" > stack.dump

warning: Can't read pathname for load map: Input/output error.
root@galera1:~/glb-1.0.1# cat stack.dump
[New LWP 13402]
[New LWP 13446]
[New LWP 13447]
[New LWP 13389]
[New LWP 13390]
[New LWP 13391]
[New LWP 13392]
[New LWP 13393]
[New LWP 13395]
[New LWP 13399]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `./src/glbd -w exec /root/glb-1.0.1/files/mysql.sh -uroot -proot -t 3 0.0.0.0:33'.
Program terminated with signal 6, Aborted.
#0 0xb7737422 in __kernel_vsyscall ()

Thread 10 (Thread 0xb4cdbb70 (LWP 13399)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb771d703 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#2 0x080519f4 in exec_thread (arg=0x947f920) at glb_wdog_exec.c:281
#3 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#4 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 9 (Thread 0xb54dcb70 (LWP 13395)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb771d703 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#2 0x080519f4 in exec_thread (arg=0x947e408) at glb_wdog_exec.c:281
#3 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#4 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 8 (Thread 0xb5cddb70 (LWP 13393)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb771d703 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#2 0x080519f4 in exec_thread (arg=0x947d1d8) at glb_wdog_exec.c:281
#3 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#4 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 7 (Thread 0xb64deb70 (LWP 13392)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb76851f6 in epoll_wait () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0x080536c0 in pool_fds_wait (pool=0xb75611a8) at glb_pool.c:261
#3 0x08054c74 in pool_thread (arg=0xb75611a8) at glb_pool.c:829
#4 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#5 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 6 (Thread 0xb6cdfb70 (LWP 13391)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb76851f6 in epoll_wait () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0x080536c0 in pool_fds_wait (pool=0xb75210f0) at glb_pool.c:261
#3 0x08054c74 in pool_thread (arg=0xb75210f0) at glb_pool.c:829
#4 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#5 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 5 (Thread 0xb74e0b70 (LWP 13390)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb76851f6 in epoll_wait () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0x080536c0 in pool_fds_wait (pool=0xb74e1038) at glb_pool.c:261
#3 0x08054c74 in pool_thread (arg=0xb74e1038) at glb_pool.c:829
#4 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#5 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 4 (Thread 0xb75ae6c0 (LWP 13389)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb764f326 in nanosleep () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0xb764f150 in sleep () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#3 0x08056b37 in main (argc=9, argv=0xbfc74754) at glb_main.c:217

Thread 3 (Thread 0xb34d6b70 (LWP 13447)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb76863f8 in accept4 () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0x08055912 in listener_thread (arg=0x9480ea0) at glb_listener.c:50
#3 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#4 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 2 (Thread 0xb3cd7b70 (LWP 13446)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb7676d96 in poll () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0x0804ea7b in ctrl_thread (arg=0x9481288) at glb_control.c:219
#3 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#4 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 1 (Thread 0xb44d8b70 (LWP 13402)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb75d9941 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0xb75dcd02 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#3 0xb7615e15 in ?? () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#4 0xb761ff01 in ?? () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#5 0xb7625acb in realloc () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#6 0x0804cd75 in glb_router_change_dst (router=0x9473d18, dst=0xb44d82e0, probe_ctx=0x947f920) at glb_router.c:280
#7 0x0805051e in wdog_collect_results (wdog=0x9473678) at glb_wdog.c:563
#8 0x080507d7 in wdog_main_loop (arg=0x9473678) at glb_wdog.c:645
#9 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#10 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

I hope, it helps,
Regards,
V.