SiriDB/siridb-server

Writes being blocked during complex query

bgreer101 opened this issue · 2 comments

bgreer101 commented

Hello,
I have noticed that when issuing a complex query, such as one that is likely to time out, it causes SiriDB to hang and blocks any writes actively occurring.

Test Setup
Write Process: writing approximately ~22k-25k values per second
Test Query Process: "select count(1h) from /wave-.*/" -> query times out of course, also not 100% sure of my syntax. Trying to test counting the number of values per 1hr for each series on all series with wave- prefix

Database
{'series': 610}
{'series_length': 1522832322}
Single node (8x32), Running on docker (latest tag), storage volume on an NVMe SSD

Order of Operations

  1. Writes ongoing
  2. Second process issues query
  3. Server cpu jumps to 100
  4. Writes all hang (siridb no longer responsive)
  5. Query times out
  6. ~2m elapses
  7. Server goes back down to usual load < 1

Questions:

  • Is the server process managing API connections single threaded?
  • Would it be possible to detect client-side timeout and halt processing of the query on the server side?
joente commented

Is the server process managing API connections single threaded?

Yes, API connections are handled by libuv (event loop) but some tasks with a request, like merging series, are done by a separate thread.

Would it be possible to detect client-side timeout and halt processing of the query on the server side?

Maybe, but I can only think of a solution where we would need to poll the status from within the task. I will take a better look at this issue to see if something can be done. It would be a good thing if long running queries could be halted.

denisgolius commented

Hello, I have noticed that when issuing a complex query, such as one that is likely to time out, it causes SiriDB to hang and blocks any writes actively occurring.

Test Setup Write Process: writing approximately ~22k-25k values per second Test Query Process: "select count(1h) from /wave-.*/" -> query times out of course, also not 100% sure of my syntax. Trying to test counting the number of values per 1hr for each series on all series with wave- prefix

Database {'series': 610} {'series_length': 1522832322} Single node (8x32), Running on docker (latest tag), storage volume on an NVMe SSD

Order of Operations

1. Writes ongoing

2. Second process issues query

3. Server cpu jumps to 100

4. Writes all hang (siridb no longer responsive)

5. Query times out

6. ~2m elapses

7. Server goes back down to usual load < 1

Questions:

* Is the server process managing API connections single threaded?

* Would it be possible to detect client-side timeout and halt processing of the query on the server side?

Sorry for of-top message but did you try VictoriaMetrics? IMAO: it can deal with these queries and with lower server requirements.

PS: as I see SiriDB looks like VictoriaMetrics but wrote on C language 🧐