[Q] Pickle metric value is None in whisper storage file.

Question

[Q] Pickle metric value is None in whisper storage file.

ashemez opened this issue 4 years ago · 10 comments

Hi,

I'm trying some pickling test with my own Python implementation and with example-pickle-client.py of carbon examples. I don't get any errors, I see my metric directory and whisper storage created but I only see None values corresponding to all the datetime values.
First of all the example-pickle-client.py sends the tuples of loadavg_1min, loadavg_5min and loadavg_15min but their values are always None and my implementation is always stored as none as well. Using epoch time seconds as the date value in my tuple as well with such a format: [('my.metric', (1608730620, 123))]

I also tested if my tuples are sent properly by implementing a socket server and sent the same tuples to this localhost server and received packages on this testing localhost server exactly how I sent. Just with size header + tuples body. I can share my code but since the example-pickle-client.py pickle values are stored None I just wanted to ask if there is any known issues?

I'm testing these all on 127.0.0.1 (localhost), running Graphite carbon-cache and carbon-relay on the localhost with the default ports. Using carbon host as 127.0.0.1 and carbon pickle port as 2004 to send the pickles. Running these all on WSL2 Ubuntu 20.04 environment.

Thanks

Answer 1 · 2020-12-23T17:02:22.000Z

Hi @ashemez ,

Dou you using same python version for sending metrics and receiving them? And which version of graphite do you use?

Theoretically, pickle protocol is not really required. You can use line protocol even from python - you can use e.g. graphyte library for that, accessible from pypi.

Answer 2 · 2020-12-23T17:30:18.000Z

Hi @deniszh,

Thanks for the response. Installed on python3.8 (using pip3) using the default installation path (https://graphite.readthedocs.io/en/stable/install-pip.html#installing-in-the-default-location) I think 1.2.0 is my carbon's version.
Yes using the same python binaries for running the metric and carbon daemons.
Thanks for the suggestion, I will test the line protocol too.

Answer 3 · 2020-12-23T18:39:34.000Z

Hi @deniszh , I just tested with graphyte library and at first I saw the values again None for those new line metrics. But then I realized that I changed the carbon.conf and relay-rules.conf previously and removed the carbon 2004 port from the destinations. I added 127.0.0.1:2004 to the destinations in relay-rules.conf and carbon.conf and restarted the daemons. Then line protocol metrics started to show-up in the graph.
I tested the pickles again but unfortunately they are still None. Maybe I'm misusing something in the configuration files.

Answer 4 · 2020-12-23T20:09:59.000Z

@ashemez : Line proto using port 2003, pickle proto using port 2004. Could you please post your configs somewhere (or here) and I'll check it?

Answer 5 · 2020-12-23T20:21:19.000Z

carbon.conf.txt
relay-rules.conf.txt

Attached my conf files, added txt extension now. Thank you.

Answer 6 · 2020-12-23T20:57:23.000Z

Are you using only single carbon-cache insance? Then you can ignore relay and relay-rules completely. Please run only carbon cache and point line proto data to port 2003 and/or pickle proto to port 2004

Answer 7 · 2020-12-23T21:12:18.000Z

Yes I'm using a single carbon-cache instance. The reason I used the relay-rules is for pointing it to Skyline instance that I intended to run at 2024 port. In carbon.conf 2003 and 2004 ports look properly set then. What is the role of 2014 port?

Answer 8 · 2020-12-23T21:58:43.000Z

Then you need to run carbon-relay.py together with carbon-cache.py and also send line proto data to port 2013 and/or pickle port data to port 2014.
Relay only routes data, cache stores it. If you send data directly to 2003/2004 (i.e. derictly to cache) it will be stored but not routed to Skyline.

Answer 9 · 2020-12-24T09:05:53.000Z

Hi @deniszh thank you for helping to figure out the ports. That makes more sense now. But I think I have another issue with the whisper file. When I'm using line protocol, if I don't pass a timestamp parameter, only put the metric name and value it starts showing up the values in the whisper file when I run this command: ./whisper-fetch.py /path/to/whisper/file.wsp
But when I add a timestamp even using the line proto it doesn't show the values in the output of ./whisper-fetch.py /path/to/whisper/file.wsp .
This output is shown like below.
...
1608800040 None
1608800100 None
1608800160 None --> Thu Dec 24 11:56:00 2020
1608800220 None --> Thu Dec 24 11:57:00 2020
...

Then I found passing the from-until parameters of whisper-fetch.py and ran like this: ./whisper-fetch.py --pretty --from=1608797400 --until=1708798300 /path/to/whisper/file.wsp this gave me the data points that I inserted in the output like below:

...
Thu Dec 24 11:55:50 2020 None
Thu Dec 24 11:56:00 2020 112.200000
Thu Dec 24 11:56:10 2020 None
Thu Dec 24 11:56:20 2020 None
Thu Dec 24 11:56:30 2020 None
Thu Dec 24 11:56:40 2020 None
Thu Dec 24 11:56:50 2020 None
Thu Dec 24 11:57:00 2020 112.200000
Thu Dec 24 11:57:10 2020 None
Thu Dec 24 11:57:20 2020 None
Thu Dec 24 11:57:30 2020 None
Thu Dec 24 11:57:40 2020 None
Thu Dec 24 11:57:50 2020 None
...

Above I added the time correspondence of the UTC times of the first output where I have values in the second command output. Not sure if this is a timezone issue because if I run the second command it shows the values.

Answer 10 · 2020-12-25T09:36:49.000Z

I guess I fixed the issue with the whisper file. I added a retention rule for my metric in the storage-schemas.conf file and it started to insert and show up with the timestamp I pass. Looks good now. Thanks again.