grafana/tempo-operator

GrafanaDatasource: support gateway and HTTPS

Opened this issue · 7 comments

andreasgerstmayr commented

Support gateway and HTTPS when creating the GrafanaDatasource:

tempo-operator/internal/manifests/grafana/datasource.go

Lines 34 to 35 in 5a79e61

URL: fmt.Sprintf("http://%s:%d", naming.ServiceFqdn(params.Tempo.Namespace, params.Tempo.Name, component), manifestutils.PortHTTPServer),
JSONData: json.RawMessage(fmt.Sprintf(`{"tlsSkipVerify": %t}`, tlsSkipVerify)),

ritaCanavarro commented

Hi @andreasgerstmayr, I would like to work on this issue could you please assign it to me?
Is there anything I need to take into account while solving it?
E.G the /ping endpoint we need for Gateway as mentioned on #423 (comment)?

  • 👍1
andreasgerstmayr commented

Hi @andreasgerstmayr, I would like to work on this issue could you please assign it to me?

Great, I've assigned it to you.

Is there anything I need to take into account while solving it? E.G the /ping endpoint we need for Gateway as mentioned on #423 (comment)?

This is already implemented: observatorium/api#575

  • 🎉1
xmj commented

How can we help move this forward?

andreasgerstmayr commented

How can we help move this forward?

Do you use the gateway in static (OIDC) or OpenShift mode?

For the static mode it should be straightforward to implement, the operator can create a datasource for each tenant and enable the "Forward OAuth identity" feature in the datasource. Note however, this will only work if you configure OAuth to log in to your Grafana instance.

For the OpenShift mode, which ServiceAccount token should be used to access the tenants? Grafana cannot be configured to use the OpenShift OAuth server to login, because the OpenShift OAuth server does not support OIDC.

xmj commented

Do you use the gateway in static (OIDC) or OpenShift mode?

Openshift, via the Multitenancy config listed in https://docs.openshift.com/container-platform/4.13/observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-configuring.html#distr-tracing-tempo-config-multitenancy_distr-tracing-tempo-configuring - we've deviated from this setup in putting Tempo, Minio and OTLP into a single namespace (openshift-distributed-tracing) and renamed the tenants, but otherwise it's fairly static.

For the OpenShift mode, which ServiceAccount token should be used to access the tenants? Grafana cannot be configured to use the OpenShift OAuth server to login, because the OpenShift OAuth server does not support OIDC.

We have a grafana-sa service-account which we already use to integrate Grafana and the Prometheus datasource. That service-account is role-bound to a tempo-reader role, so I'd expect it to work.

The only things I don't yet know:

  • How do I pass the X-Scope-OrgID of the tenant to Tempo?
  • Which suffix do I use with the tempo-<instance-name>-gateway.openshift-distributed-tracing.svc.cluster.local:8080 URL through the gateway?

What I've also tried was passing grafana.createDatasource: true to the instance, until I saw the feature-gate hint and the warnings in ArgoCD.

andreasgerstmayr commented

For the OpenShift mode, which ServiceAccount token should be used to access the tenants? Grafana cannot be configured to use the OpenShift OAuth server to login, because the OpenShift OAuth server does not support OIDC.

We have a grafana-sa service-account which we already use to integrate Grafana and the Prometheus datasource. That service-account is role-bound to a tempo-reader role, so I'd expect it to work.

That'll give everyone who can access Grafana (read) access to all tenants? It does work technically, but I don't think it's great from a security point of view.

The only things I don't yet know:

  • How do I pass the X-Scope-OrgID of the tenant to Tempo?

  • Which suffix do I use with the tempo-<instance-name>-gateway.openshift-distributed-tracing.svc.cluster.local:8080 URL through the gateway?

For the ingest path I suggest OTEL, described in the link above.
For the Tempo API, it's https://<route-to-gateway>/api/traces/v1/<tenantName>/tempo

In the Grafana Tempo docs there's a page on how to setup the datasource manually: https://grafana.com/docs/tempo/latest/setup/operator/grafana_datasource/#manual-data-source-configuration

xmj commented

For the OpenShift mode, which ServiceAccount token should be used to access the tenants? Grafana cannot be configured to use the OpenShift OAuth server to login, because the OpenShift OAuth server does not support OIDC.

We have a grafana-sa service-account which we already use to integrate Grafana and the Prometheus datasource. That service-account is role-bound to a tempo-reader role, so I'd expect it to work.

That'll give everyone who can access Grafana (read) access to all tenants? It does work technically, but I don't think it's great from a security point of view.

Understood, but that's what we have rbac for ... in Grafana ;-)

The only things I don't yet know:

  • How do I pass the X-Scope-OrgID of the tenant to Tempo?
  • Which suffix do I use with the tempo-<instance-name>-gateway.openshift-distributed-tracing.svc.cluster.local:8080 URL through the gateway?

For the ingest path I suggest OTEL, described in the link above. For the Tempo API, it's https://<route-to-gateway>/api/traces/v1/<tenantName>/tempo

In the Grafana Tempo docs there's a page on how to setup the datasource manually: https://grafana.com/docs/tempo/latest/setup/operator/grafana_datasource/#manual-data-source-configuration

Sweet - we now have this working using the GrafanaDatasource with the Tempo Gateway behind an HTTPS endpoint.

I'm attaching the GrafanaDatasource in case someone can make good use on OpenShift (while noting the security caveats), maybe we can come up with some better solution in the future ;-)

tempodatasource.yaml.txt

  • 🚀1