SiriDB/siridb-server

Possible endless loop on point insert

m1ch4ls opened this issue · 2 comments

We have encountered one of our instances spinning 100% CPU. The instance was unavailable (couldn't accept connections) so I assume it entered some kind of endless loop. I took a core dump before restart.

SiriDB Server 2.0.38
libcleri 0.12.1
Core

#0  0x000055ebe4c1986c in siridb_points_add_point ()
#1  0x000055ebe4c23139 in siridb_series_add_point ()
#2  0x000055ebe4c062da in INSERT_local_task ()
#3  0x00007f64f47dd7b3 in ?? () from /home/ubnt/tmp/libuv.so.1
#4  0x00007f64f47eb43b in uv.io_poll () from /home/ubnt/tmp/libuv.so.1
#5  0x00007f64f47ddd7a in uv_run () from /home/ubnt/tmp/libuv.so.1
#6  0x000055ebe4c362fa in siri_start ()
#7  0x000055ebe4becd53 in main ()

Other threads were doing nothing

  Id   Target Id         Frame 
* 1    LWP 6489          0x000055ebe4c1986c in siridb_points_add_point ()
  2    LWP 6690          0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
  3    LWP 6742          0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
  4    LWP 9732          0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
  5    LWP 9733          0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
  6    LWP 9734          0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
  7    LWP 9735          0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
  8    LWP 9736          0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
  9    LWP 9737          0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
  10   LWP 9738          0x00007f64f4849c4f in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
  11   LWP 9739          0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1

Thread 11 (LWP 9739):
#0  0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#1  0x00007f64f4849c6d in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#2  0x00007f64efdd5b64 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 10 (LWP 9738):
#0  0x00007f64f4849c4f in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#1  0x0000000000000006 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 9 (LWP 9737):
#0  0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#1  0x00007f64f4849c6d in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#2  0x00007f64f0ddbb64 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 8 (LWP 9736):
#0  0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#1  0x00007f64f4849c6d in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#2  0x00007f64f15deb64 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 7 (LWP 9735):
#0  0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#1  0x00007f64f4849c6d in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#2  0x00007f64f1de1b64 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 6 (LWP 9734):
#0  0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#1  0x00007f64f4849c6d in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#2  0x00007f64f25e4b64 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 5 (LWP 9733):
#0  0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#1  0x00007f64f4849c6d in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#2  0x00007f64f2de7b64 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 4 (LWP 9732):
#0  0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#1  0x00007f64f4849c6d in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#2  0x00007f64f35eab64 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 3 (LWP 6742):
#0  0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#1  0x00007f64f4849c6d in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#2  0x00007f64f3dedb64 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 2 (LWP 6690):
#0  0x00007f64f484c878 in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#1  0x00007f64f4849c6d in ?? () from /home/ubnt/tmp/lib/ld-musl-x86_64.so.1
#2  0x00007f64f464eb64 in ?? ()
#3  0x0000000000000000 in ?? ()

Btw: this is our own Docker image based on Alpine Linux 3.12. I can, of course, share the image if necessary.

@m1ch4ls , I'm not 100% sure but the add point call made a dangerous assumption which is removed in the master branch. Can you build an image from the master branch and check if the issue is resolved? (should be at least version 2.0.41-alpha-0)

@joente So far no more problems. The server has been stable for a week. Thanks for the quick fix.