istio/ztunnel

metrics endpoint sends content-type: application/openmetrics-text

Closed this issue · 9 comments

craigbox commented

Could ztunnel serve /metrics as text/plain if the user hasn't explicitly sent an Accept: header for application/openmetrics-text? Without, the browser will download the file, rather than showing it as plain text on the screen.

spring-projects/spring-boot#28446 is an example of this pattern being fixed in Spring Boot.

(A waypoint, OTOH, will return text/plain and work as expected)

howardjohn commented

Envoy: content-type: text/plain; charset=UTF-8
Istio-agent: Content-Type: text/plain; version=0.0.4; charset=utf-8
Ztunnel: content-type: application/openmetrics-text;charset=utf-8;version=1.0.0

So yeah this looks right && there is a bug in istio-agent as well putting version when it shouldn't

ilrudie commented

This definitely is fixable, just need to decide on the correct behavior. Based on the above from John, do we care to ever have the openmetrics-text content type or just always say its plain text?

howardjohn commented
  • 👍1
howardjohn commented
  • 👍1
ilrudie commented

In testing, near as I can tell envoy ignores the accept header entirely. You have 2 different content-types being returned which are based on query params, not headers. If you include ?format=json you get 'application/json' and for everything else you get 'text/plain'. This assumes you are querying the internal admin interface which has more options. If you use 15090/stats/prometheus then it always uses 'text/plain' even if you request json formatting.

howardjohn commented

Well that might break with prometheus 3.0! prometheus/prometheus#15136

Note we don't always hit envoy directly but rather proxy through istio-agent, so that one may matter less

ilrudie commented

Interesting. I think envoy behavior is probably fine for this. They call out that text/plain is the content-type expected for Prom parser and Envoy specifically supports Prom format, but does not support OpenMetrics (which is similar but stricter from what I can gather).

howardjohn commented

Ztunnel is more strictly openmetrics fwiw

  • 👍1
ilrudie commented

Perfect. For ztunnel we can have the "if you request openmetrics with Accept you get openmetrics back" behavior then. I have some code in the works but it is not pushed anywhere yet.