go-graphite/graphite-clickhouse

Prometheus query does not work for empty labels

Rast1234 opened this issue · 5 comments

Rast1234 commented

Prometheus itself supports querying for absent labels, eg. metric{label=""}. However, it does not work here with prometheus handler, it returns empty data.

Felixoid commented

Sorry, but it works for me with the latest master

image

This is my data in graphite.tagged

:) SELECT * FROM graphite.tagged

SELECT *
FROM graphite.tagged

┌───────Date─┬─Tag1──────────┬─Path───────┬─Tags──────────────────────┬────Version─┐
│ 2021-03-02 │ __name__=test │ test?tag1= │ ['__name__=test','tag1='] │ 1614713518 │
│ 2021-03-02 │ tag1=         │ test?tag1= │ ['__name__=test','tag1='] │ 1614713518 │
└────────────┴───────────────┴────────────┴───────────────────────────┴────────────┘
Rast1234 commented

Interesting. You store tag1 with empty value, right? What if tag is absent in storage? Prometheus treats absent values as empty, maybe Graphite/Clickhouse does not, i am not expert in this...
Eg. in your example does this query return data? test{tag2=""}

Felixoid commented

No, it does not. It explicitly searches for the tag in the table:

SELECT Path FROM graphite.tagged WHERE ((Tag1='__name__=test') AND (arrayExists((x) -> x='tag2=', Tags))) AND (Date >='2021-03-02' AND Date <= '2021-03-02') GROUP BY Path

Rast1234 commented

That's it! :) From Prometheus point of view it should work...

My use case: dashboards in Grafana, datasources are prometheus and graphite-prometheus. Grafana queries use variables, they can not be altered (eg. remove label from query conditionally). Sometimes i want to narrow results by a label:

test{label=~"var"}

And sometimes i need to see all the data, so variable becomes empty:

test{label=~""}

Graphite-prometheus does not show data which does not have this label at all. Prometheus returns everything.

EDIT: i forgot regex operators =~. but still issue is about = because it does not work as expected too...

Felixoid commented

I've just read the doc of graphite-web, where an empty tag is not allowed anymore since 1.1.6 https://graphite.readthedocs.io/en/latest/tags.html

I'm not sure yet, how to handle it properly. The carbon-clickhouse and graphite-clickhouse should be heavily adjusted to the current state, it looks like.

And I'm not sure, if test{label=""} can be treated as any data. I'd expect it more like tag is not exist for the metric

In any case, you can jump into the discussion in group https://t.me/ru_go_graphite/1827

  • 👀1