New metric since logstash 8.13: worker_utilization

[VCCPlindsten]

Feature Description

There is a new metric on pipeline level since 8.13: https://www.elastic.co/guide/en/logstash/current/logstash-8-13-0.html#notable-8.13.0

Please collect and include it.

_node/stats sample (logstash 8.15.0)

"main": {
      "events": {
        "duration_in_millis": 1544425,
        "filtered": 4510626,
        "out": 4510626,
        "in": 4510877,
        "queue_push_duration_in_millis": 1269044
      },
      "flow": {
        "input_throughput": {
          "current": 1765,
          "last_1_minute": 2083,
          "last_5_minutes": 2111,
          "last_15_minutes": 1754,
          "lifetime": 1487
        },
        "worker_concurrency": {
          "current": 0.5364,
          "last_1_minute": 0.6848,
          "last_5_minutes": 0.6716,
          "last_15_minutes": 0.5716,
          "lifetime": 0.5093
        },
        "filter_throughput": {
          "current": 1765,
          "last_1_minute": 2079,
          "last_5_minutes": 2111,
          "last_15_minutes": 1754,
          "lifetime": 1487
        },
        "output_throughput": {
          "current": 1765,
          "last_1_minute": 2079,
          "last_5_minutes": 2111,
          "last_15_minutes": 1754,
          "lifetime": 1487
        },
        "worker_utilization": {
          "current": 53.64,
          "last_1_minute": 68.48,
          "last_5_minutes": 67.16,
          "last_15_minutes": 57.16,
          "lifetime": 50.93
        },
        "queue_backpressure": {
          "current": 0.4336,
          "last_1_minute": 0.5846,
          "last_5_minutes": 0.5729,
          "last_15_minutes": 0.4807,
          "lifetime": 0.4185
        }
      },

Version of logstash-exporter this feature request applies to

None

Motivation and Goals

This is a useful metric to determine how saturated logstash is since it uses the pipeline-level configured worker count, 100% means its saturated, no matter how many workers are currently configured.

Alternatives Considered

Using a different way to collect logstash metrics - not a good option.

Additional Context

There doesn't seem to be another way to get at this number - because the amount of workers is a per-pipeline attribute, which doesn't seem to be exposed by logstash.
logstash_info_pipeline_workers only collects the default number.

[kuskoman]

@VCCPlindsten are you using v1, or v2?
I create a PR for v1: #369
but v2 is in rather unstable state on the branches that i am working on RN and i wonder whether i have to rebase them, or just fix that later on

[kuskoman]

also, i forgot to ask. could you do the review on the PR i just posted?if it is allright i can merge it soon and create a PR

[VCCPlindsten]

Looking a little bit closer at the _node api, it seems per-pipeline workers is exposed, under the "pipelines" key. Not sure if that is a new thing as well - I've not noticed it before.

Reviewing shortly.

[kuskoman]

i am currently working and should not be doing this task, but it seems like a small one :D
the PR was created really quick, so there may be these mistakes in it

Are you using v1 version of the exporter? I will replicate the PR to v2 later on

[VCCPlindsten]

I am using v1, yes - since v2 is not stable yet.

[VCCPlindsten]

Working in release 1.8.2