akumuli/Akumuli

Akumuli crashing on query

sptrakesh opened this issue · 2 comments

I am running via the latest Docker image (akumuli:0.8.80-skylake). The following query via the Grafana plugin causes the container to crash.

spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,816 http [INFO] Cursor 140025295594008 created
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,816 http [INFO] Cursor 140025295594008 started
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,817 Main [INFO] Parsing query:
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,818 Main [INFO] {
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "select": "request.counter",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "range": {
spt_akumuli.1.sullcoizlnqg@docker-desktop    |         "from": "20200605T230206.779",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |         "to": "20200605T200206.779"
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     },
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "where": "",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "order-by": "series",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "apply": "",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "limit": "1000000",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "pivot-by-tag": [
spt_akumuli.1.sullcoizlnqg@docker-desktop    |         "resource"
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     ]
spt_akumuli.1.sullcoizlnqg@docker-desktop    | }
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,819 Main [ERROR] MergeIterator - broken invariant
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,820 Main [ERROR] Terminating (core dumped)

Akumuli logs also show this error:

2020-06-05 23:09:30,277 Main [INFO] 18446744073709550400 Trying to open tree, repair status - OK, addr: 20537
2020-06-05 23:09:30,278 Main [ERROR] Empty rescue points list found, leaf-node data was lost
2020-06-05 23:09:30,278 Main [ERROR] Repair needed, id=1145
Lazin commented

How your tags look like? Is 'resource' the only tag in the series name?

No, there are additional tags such as method, resource, ipaddress, status and mimeType.

Sorry, an OS update wiped out the old logs. I was able to trace the issue back further to what looked like a database corruption (probably caused when writing the series). The query causing the crash seemed to be an after effect of that corruption.

Will create another issue if I run into it again. Until then probably only option is close this issue.