metrics not available through fleet-manager endpoint
k-wall opened this issue · 2 comments
Attempts to read the metrics currently fail within the fleet-manager. This occurs because there is no observability component in the kas-installer installed environment.
Internally, fleet-manager fails like this:
E0715 10:17:40.368875 1 api.go:262] error from metric Post "/api/metrics/v1/test/api/v1/query": can't request metrics without auth
E0715 10:17:40.368900 1 metrics.go:55] error getting metrics: KAFKAS-MGMT-9: failed to retrieve metrics
and returns 500 back to the caller.
Discussions with @pb82 and @JameelB initially had us explored configuring fleetmanager to point directly at the prometheus end-point, however, this doesn't work:
the Observatorium tenant is appended to the URL. the URL Observatorium specific and not compatible with raw Prometheus
https://github.com/pb82/kas-fleet-manager/blob/main/pkg/client/observatorium/client.go#L52
One option is to introduce some special flags into fleet-manager giving it the ability to talk directly to a prometheus. Alternatively, perhaps observatorium could be deployed by the kas-installer.
@MikeEdgar pointed out to me that kas-installer does have the capability to install observatorium. I'll give this a go and experiment with wiring fleetmanager to it.