Service Monitor should optionally also scrape port 8090 for process metrics

[sarasensible]

Is your feature request related to a problem? Please describe.
We have monitoring of varnish-specific metrics but not process metrics like cpu and memory.

Describe the solution you'd like
A simple change to the helm chart should enable it.

Describe alternatives you've considered
I can create an additional ad-hoc service monitor.

Additional context
I'll create a PR for this.

[sarasensible]

I'm starting to suspect my solution isn't quite what I'd hoped - the process metrics from the signaller don't seem to reflect the varnish pod's resource usage.

Here you can see the process metrics hovering around 34 MB and a slight fraction of CPU:

However the Varnish cache size is set to 8G and I believe most of that is being used. So these process metrics may only reflect the signaller.

Is there a way to capture the varnish process metrics?

[martin-helmich]

Out of curiosity, what are the actual metrics that you are visualising in your screenshots? AFAIK, the signaller /metrics endpoint should only return signaller-relevant metrics (and maybe at most the built-in metrics provided by the Go-runtime); in any case, you should not be able to receive any relevant Pod utilisation metrics that way.

For the regular Pod resource utilisation metrics, I'd recommend relying on metrics provided by Kubernetes itself, like the cAdvisor metrics published by the kubelet.

[sarasensible]

Hi @martin-helmich , thanks for your response! I am using the following queries:
process_resident_memory_bytes{job="sensible-cache"} and
rate(process_cpu_seconds_total{job="sensible-cache"}[1m])

I can definitely do that, I was hoping the exporter container would be a "one stop shop" but I can definitely work around it in the way you suggest. As an aside the exporter project looks somewhat abandoned so I don't think there's much hope in asking that maintainer to add this as a feature. I'll close the PR since signaller metrics aren't particularly interesting for our use case. Thanks!

[martin-helmich]

I am using the following queries: [...]

Yeah... Those two will only give you the RSS and CPU usage of the Go process itself, but not of any subprocesses running within the same container (such as the varnishd subprocesses).

[...] so I don't think there's much hope in asking that maintainer to add this as a feature.

In general, I would assume that no application-specific exporter will export generic container utilisation metrics (especially since you basically get those "for free" since the kubelet provides these metrics anyway).

That being said, since the signaller does indeed export metrics (albeit not those you expected), it would still make sense to include the signaller's metrics endpoint into the ServiceMonitor; so you're welcome to leave #170 open if you like. 😉

[sarasensible]

Fair enough - reopened!