How to let graphite-web's tag works with graphite-clickhouse?

Question

How to let graphite-web's tag works with graphite-clickhouse?

Closed this issue 4 years ago · 20 comments

I can get tag data from graphite-clickhouse using command `curl 'http://127.0.0.1:9090/tags/autoComplete/tags?pretty=1', and also get data from graphite-web in the same way. But I cannot see my remote agent's host information in graphite-web metrics section.
graphite-web == 1.1.7

Answer 1 · 2021-03-08T09:54:03.000Z

Hey. GCH uses the tags written by carbon-clickhouse to the tagged table, see https://github.com/lomik/carbon-clickhouse, search for tagged on the page. I'm not quite sure, how to set up the external tags for graphite-web, since use carbonapi.

Maybe @deniszh can help here.

Answer 2 · 2021-03-08T10:06:06.000Z

@leavesbnw : set in local_settings.py

TAGDB_HTTP_URL = 'https://127.0.0.1:9090'
TAGDB_HTTP_AUTOCOMPLETE = True

Answer 3 · 2021-03-08T10:11:44.000Z

= True

I set those, maybe I didn't make it clear. When I didn't use tag, I can see remote agent host information in the metrics tree, but I didn't can see it after I configure tag.

Answer 4 · 2021-03-08T10:17:01.000Z

When I configure tag, my data in graphite table look like this below:
kw.kw :) select Path from graphite where Path like '%172.17.76.29%' group by Path limit 10;

SELECT Path
FROM graphite
WHERE Path LIKE '%172.17.76.29%'
GROUP BY Path
LIMIT 10

┌─Path────────────────────────────────────────────────────────────────────────────────────────────┐
│ mem.huge_pages_total?dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │
│ diskio.weighted_io_time?_name=sr0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │
│ diskio.write_bytes?_name=sr0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │
│ disk.inodes_free?dc=yf&device=dm-0&fstype=xfs&host=172.17.76.29&mode=rw&path=-&rack=B6-L4-1-D05 │
│ diskio.read_time?_name=dm-1&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │
│ diskio.merged_reads?_name=dm-1&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │
│ diskio.reads?_name=sr0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │
│ diskio.write_bytes?_name=dm-0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │
│ diskio.write_bytes?_name=sda2&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │
│ mem.low_free?dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │
└─────────────────────────────────────────────────────────────────────────────────────────────────┘

Answer 5 · 2021-03-08T10:28:28.000Z

Sorry, but a name like mem.huge_pages_total?dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 doesn't look like tags to me.

The graphite tags are supposed to be mem.huge_pages_total;dc=yf;host=172.17.76.29;rack=B6-L4-1-D05

Then they will be written to the dedicated table.

Answer 6 · 2021-03-08T11:35:31.000Z

Sorry, but a name like mem.huge_pages_total?dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 doesn't look like tags to me.

The graphite tags are supposed to be mem.huge_pages_total;dc=yf;host=172.17.76.29;rack=B6-L4-1-D05

Then they will be written to the dedicated table.

I don't know why, my architecture is telegraf-->carbon-clickhouse-->clickhouse--->graphite-clickhouse-->graphite-web.

Answer 7 · 2021-03-08T11:46:50.000Z

I use tcp mode let telegraf send data to carbon-clickhouse

Answer 8 · 2021-03-08T12:06:14.000Z

It might be worth to test different version of telegraf as well (either get previous major release or newest in case yours is not the most recent)

Answer 9 · 2021-03-08T12:12:55.000Z

version information:
telegraf: v1.16.2
carbon-clickhouse: v0.11.1
graphite-clickhouse: v0.12.0
graphite-web: v1.1.7

Answer 10 · 2021-03-08T13:54:27.000Z

@leavesbnw : do you have graphite tags enabled in Telegraf?
https://docs.influxdata.com/telegraf/v1.17/data_formats/output/graphite/

Answer 11 · 2021-03-09T02:12:47.000Z

@leavesbnw : do you have graphite tags enabled in Telegraf?
https://docs.influxdata.com/telegraf/v1.17/data_formats/output/graphite/

Yes, I have enabled tag in Telegraf.

Answer 12 · 2021-03-09T02:20:06.000Z

telegraf.conf:
[[outputs.graphite]]
  servers = ["172.17.76.41:2003"]
  prefix = ""
  template = "host.tags.measurement.field"
  graphite_tag_support = true
  graphite_separator = "."
  timeout = 2

Answer 13 · 2021-03-09T07:41:22.000Z

I have tested telegraf, telegraf send data like 'mem.huge_pages_total;dc=yf;host=172.17.76.29;rack=B6-L4-1-D05 0 1615275010' to carbon-clickhouse, but carbon-clickhouse saved this data in tmp file /tmp/carbon-clickhouse/default.xxxxxxxxx.lz4 using format 'mem.huge_pages_total?dc=yf&host=172.17.76.29&rack=B6-L4-1-D05'.Is there any wrong configuration for me in carbon-clickhouse?
my carbon-clickhouse.conf is:

[common]
metric-prefix = "carbon.agents.{host}"
metric-endpoint = "local"
metric-interval = "1m0s"
max-cpu = 1
[logging]
file = "/var/log/carbon-clickhouse/carbon-clickhouse.log"
level = "debug"
[data]
path = "/tmp/carbon-clickhouse/"
chunk-max-size = 0
chunk-interval = "5s"
chunk-auto-interval = "5:10s,20:60s"
compression = "lz4"
compression-level = 0
[upload.graphite]
type = "points-reverse"
table = "graphite.graphite"
threads = 1
url = "http://localhost:8123/"
compress-data = true
timeout = "1m0s"
zero-timestamp = true
[upload.graphite_index]
type = "index"
table = "graphite.graphite_index"
threads = 1
url = "http://localhost:8123/"
timeout = "1m0s"
cache-ttl = "12h0m0s"
hash = ""
disable-daily-index = false
[upload.graphite_tagged]
type = "tagged"
table = "graphite.graphite_tagged"
threads = 1
url = "http://localhost:8123/"
timeout = "1m0s"
cache-ttl = "12h0m0s"
[udp]
listen = ":2003"
enabled = true
drop-future = "0s"
drop-past = "0s"
drop-longer-than = 0
[tcp]
listen = ":2003"
enabled = true
drop-future = "0s"
drop-past = "0s"
drop-longer-than = 0
[pickle]
listen = ":2004"
enabled = true
drop-future = "0s"
drop-past = "0s"
drop-longer-than = 0
[grpc]
listen = ":2005"
enabled = false
drop-future = "0s"
drop-past = "0s"
drop-longer-than = 0
[prometheus]
listen = ":2006"
enabled = false
drop-future = "0s"
drop-past = "0s"
drop-longer-than = 0
[telegraf_http_json]
listen = ":2007"
enabled = true
drop-future = "0s"
drop-past = "0s"
drop-longer-than = 0
concat = "."
[pprof]
listen = "localhost:7007"
enabled = false

Answer 14 · 2021-03-09T07:43:45.000Z

Do you send it to :2003 or to :2007? If you send the carbon plain text format, it should land on the port 2003

upd: I see, ["172.17.76.41:2003"]

Answer 15 · 2021-03-09T07:44:24.000Z

Do you send it to :2003 or to :2007? If you send the carbon plain text format, it should land on the port 2003

2003

Answer 16 · 2021-03-09T07:55:36.000Z

type = "points-reverse"

This should revert the metrics, but it doesn't look so. The metric mem.huge_pages_total should become huge_pages_total.mem.

Just to exclude some fixed problems, can you try to use the master? You may find the package in the workflow https://github.com/lomik/carbon-clickhouse/runs/1936087709 artifacts, direct link

Answer 17 · 2021-03-09T08:21:33.000Z

points-reverse

I used the .deb package, the result is the same as before. clickhouse data is like this beblow:

kw.kw :) select * from graphite where Path like '%172.17.76.29%' limit 10;

SELECT *
FROM graphite
WHERE Path LIKE '%172.17.76.29%'
LIMIT 10

┌─Path────────────────────────────────────────────────────────────────────────┬─────────────Value─┬───────Time─┬───────Date─┬─Timestamp─┐
│ cpu.usage_guest?cpu=cpu-total&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05      │                 0 │ 1615277790 │ 2021-03-09 │         0 │
│ cpu.usage_guest?cpu=cpu0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05           │                 0 │ 1615277790 │ 2021-03-09 │         0 │
│ cpu.usage_guest_nice?cpu=cpu-total&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │                 0 │ 1615277790 │ 2021-03-09 │         0 │
│ cpu.usage_guest_nice?cpu=cpu0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05      │                 0 │ 1615277790 │ 2021-03-09 │         0 │
│ cpu.usage_idle?cpu=cpu-total&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05       │ 99.19028340240304 │ 1615277790 │ 2021-03-09 │         0 │
│ cpu.usage_idle?cpu=cpu0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05            │ 99.19028340240304 │ 1615277790 │ 2021-03-09 │         0 │
│ cpu.usage_iowait?cpu=cpu-total&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05     │                 0 │ 1615277790 │ 2021-03-09 │         0 │
│ cpu.usage_iowait?cpu=cpu0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05          │                 0 │ 1615277790 │ 2021-03-09 │         0 │
│ cpu.usage_irq?cpu=cpu-total&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05        │                 0 │ 1615277790 │ 2021-03-09 │         0 │
│ cpu.usage_irq?cpu=cpu0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05             │                 0 │ 1615277790 │ 2021-03-09 │         0 │
└─────────────────────────────────────────────────────────────────────────────┴───────────────────┴────────────┴────────────┴───────────┘

kw.kw :) select Path, Tags from graphite_tagged where Path like '%172.17.76.29%' limit 10;

SELECT
    Path,
    Tags
FROM graphite_tagged
WHERE Path LIKE '%172.17.76.29%'
LIMIT 10

┌─Path────────────────────────────────────────────────────────────────────────┬─Tags─────────────────────────────────────────────────────────────────────────────────────────────┐
│ cpu.usage_guest?cpu=cpu-total&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05      │ ['__name__=cpu.usage_guest','host=172.17.76.29','rack=B6-L4-1-D05','cpu=cpu-total','dc=yf']      │
│ cpu.usage_guest?cpu=cpu0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05           │ ['__name__=cpu.usage_guest','cpu=cpu0','dc=yf','host=172.17.76.29','rack=B6-L4-1-D05']           │
│ cpu.usage_guest_nice?cpu=cpu-total&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05 │ ['__name__=cpu.usage_guest_nice','cpu=cpu-total','dc=yf','host=172.17.76.29','rack=B6-L4-1-D05'] │
│ cpu.usage_guest_nice?cpu=cpu0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05      │ ['__name__=cpu.usage_guest_nice','host=172.17.76.29','rack=B6-L4-1-D05','cpu=cpu0','dc=yf']      │
│ cpu.usage_idle?cpu=cpu-total&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05       │ ['__name__=cpu.usage_idle','host=172.17.76.29','rack=B6-L4-1-D05','cpu=cpu-total','dc=yf']       │
│ cpu.usage_idle?cpu=cpu0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05            │ ['__name__=cpu.usage_idle','cpu=cpu0','dc=yf','host=172.17.76.29','rack=B6-L4-1-D05']            │
│ cpu.usage_iowait?cpu=cpu-total&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05     │ ['__name__=cpu.usage_iowait','host=172.17.76.29','rack=B6-L4-1-D05','cpu=cpu-total','dc=yf']     │
│ cpu.usage_iowait?cpu=cpu0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05          │ ['__name__=cpu.usage_iowait','cpu=cpu0','dc=yf','host=172.17.76.29','rack=B6-L4-1-D05']          │
│ cpu.usage_irq?cpu=cpu-total&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05        │ ['__name__=cpu.usage_irq','host=172.17.76.29','rack=B6-L4-1-D05','cpu=cpu-total','dc=yf']        │
│ cpu.usage_irq?cpu=cpu0&dc=yf&host=172.17.76.29&rack=B6-L4-1-D05             │ ['__name__=cpu.usage_irq','dc=yf','host=172.17.76.29','rack=B6-L4-1-D05','cpu=cpu0']             │
└─────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘

Are you sure the "Path" data in clickhouse is the format 'cpu.usage_guest;cpu=cpu-total;dc=yf;host=172.17.76.29;rack=B6-L4-1-D05' or did I get it wrong in the first place?

Answer 18 · 2021-03-09T09:39:09.000Z

Sorry, it's totally my bad. The format is correct as I just checked from my CH

SELECT *
FROM tagged
WHERE Path = 'test.ms_shiryaev.name?tag1=12&tag2=tag'

┌───────Date─┬─Tag1───────────────────────────┬─Path───────────────────────────────────┬─Tags────────────────────────────────────────────────────┬────Version─┐
│ 2021-03-02 │ __name__=test.ms_shiryaev.name │ test.ms_shiryaev.name?tag1=12&tag2=tag │ ['__name__=test.ms_shiryaev.name','tag1=12','tag2=tag'] │ 1614713245 │
│ 2021-03-02 │ tag1=12                        │ test.ms_shiryaev.name?tag1=12&tag2=tag │ ['__name__=test.ms_shiryaev.name','tag1=12','tag2=tag'] │ 1614713245 │
│ 2021-03-02 │ tag2=tag                       │ test.ms_shiryaev.name?tag1=12&tag2=tag │ ['__name__=test.ms_shiryaev.name','tag1=12','tag2=tag'] │ 1614713245 │
└────────────┴────────────────────────────────┴────────────────────────────────────────┴─────────────────────────────────────────────────────────┴────────────┘

SELECT *
FROM data
WHERE Path = 'test.ms_shiryaev.name?tag1=12&tag2=tag'

┌─Path───────────────────────────────────┬─Value─┬───────Time─┬───────Date─┬──Timestamp─┐
│ test.ms_shiryaev.name?tag1=12&tag2=tag │   123 │ 1614643200 │ 2021-03-02 │ 1614713243 │
└────────────────────────────────────────┴───────┴────────────┴────────────┴────────────┘

And as well as for you, they are shown by the api

curl 'http://127.0.0.1:9090/tags/autoComplete/tags?pretty=1'
["name","tag1","tag2"]

So, your ClickHouse part works correctly.

But I cannot see my remote agent's host information in graphite-web metrics section.

Do you mean here, that you don't see requests from graphite-web or what?

When I didn't use tag, I can see remote agent host information in the metrics tree, but I didn't can see it after I configure tag.

Or that you don't see other tags for the metric anymore?

Answer 19 · 2021-03-09T09:50:28.000Z

@leavesbnw :

But I cannot see my remote agent's host information in graphite-web metrics section.

You mean metrics with tags are not visible in UI? Yes, that's a downside. Graphite-web is old and not designed for tags. You should use functions to select metrics by tags, as described in https://graphite.readthedocs.io/en/latest/tags.html#querying or use Grafana.
Metric itself should be visible, but tags are not. If metrics is not visible even - that's something is wrong, indeed.

Answer 20 · 2021-03-09T09:54:15.000Z

@leavesbnw :

But I cannot see my remote agent's host information in graphite-web metrics section.

You mean metrics with tags are not visible in UI? Yes, that's a downside. Graphite-web is old and not designed for tags. You should use functions to select metrics by tags, as described in https://graphite.readthedocs.io/en/latest/tags.html#querying or use Grafana.
Metric itself should be visible, but tags are not. If metrics is not visible even - that's something is wrong, indeed.

Yes, I want metrics with tags are visible in UI.Thank you,Thank you very much for your help and answer.Now I am trying another architecure:telegraf-->carbon-clickhouse-->clickhouse-->graphite-clickhouse-->carbonapi-->grafana.