hyperledger/indy-node-monitor

Not sure why `response_result_data_Hardware_HDD_used_by_node` is used as a label in prometheus metrics

pruneau628 opened this issue · 12 comments

pruneau628 commented

It artificially creates different times series, is there a rationale for this ?

WadeBarnes commented

There are a number of text fields returned by the call to validator-info that we want to display. Prometheus does not handle text, so adding the text fields to labels allows the text to be accessed in Prometheus queries. This was done when we were initially evaluating Prometheus and Influx.

pruneau628 commented

I do understand the purpose, and I can get behind including attributes that seldom changes, like IP configuration, OS version, but the disk space usage change a bit too often to qualify as a label, though. It also changes randomly, which makes it even worse as a label.

WadeBarnes commented

It would be better to handle text fields through Influx and remove these labels from Prometheus, which brings up a question. Do we use Prometheus and Influx, or just Influx? Prometheus makes collecting and graphing metrics super easy, but it does not handle text. Influx is more difficult when it comes to queries, but supports text and metrics. Or is there another solution? @ConnorBarnes88, do you have any thoughts on this?

pruneau628 commented
  • from a restricted point of view, response_result_data_Hardware_HDD_used_by_node is actually a number, and should be treated as such
  • removing prometheus from the equation creates a void with respect of raising alarms.
  • it could also make sense to use this monitoring stack to gather metrics from agents and other parts of the eosystem, and prometheus is still a good candidate.

Now, I have the feeling I'm missing the big picture here.

WadeBarnes commented
  • from a restricted point of view, response_result_data_Hardware_HDD_used_by_node is actually a number, and should be treated as such

The issue is we receive it as text in human readable form where the number could be in MBs, GBs, etc:

"Hardware": {
  "HDD_used_by_node": "1037 MBs"
},

Converting that into an actual number will require some transformation as the telegraf level. Converting it to a number would allow us to remove it as a label; stop treating it as text.

  • removing prometheus from the equation creates a void with respect of raising alarms.
  • it could also make sense to use this monitoring stack to gather metrics from agents and other parts of the eosystem, and prometheus is still a good candidate.

Agreed

Now, I have the feeling I'm missing the big picture here.

I don't think you are.

pruneau628 commented

Thanks for your trust @WadeBarnes.
I have been looking into my own affirmation about prometheus, and it actually looks like the latest influxdb oss version 2.7.0 support alarms as well.
If prometheus is still perceived as useless, this is probably the way to go.
But that's just from reading the docs, not experimentation yet.

ConnorBarnes88 commented
pruneau628 commented

Wow, I was not expecting a simple question to devovle into a full-on architecture review.
I'm definitely interested in contributing, but not sure how to go about it.

WadeBarnes commented

lol, you picked the right (or wrong) question @pruneau628. For this metric I think we could do a conversion so Prometheus can consume it more easily. However there are many other data elements from the validator-info calls that we're interested in and are in text form. So we still have a few choices to make. Stick with a mix of Prometheus and Influx and use each for their strengths, or switch to Influx completely. Both choices introduce a certain level of complexity, the question is which will be easier to maintain in the log run.

Perhaps you could assist with the decisions by doing a bit more investigation into the capabilities of Influx 2.x.

pruneau628 commented

Ok, we are doing just that, and will report on the progress as soon as possible.

  • 👍1