Support for service account authentication

Question

Support for service account authentication

and-mora opened this issue 8 months ago · 16 comments

ASIS

As far as i understand, the only supported authentication method is the session cookie which usually happens through web browser.

Use case

I want to generate a report through API calls with alternative auth methods (like service account).

Problem

I get a 401 from the plugin if i authenticate against the grafana server with basic auth or service account (see the log from the plugin).

logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-04-16T07:11:21.228238372Z level=error msg="error generating report" endpoint=callResource err="error fetching dashboard **uid**: error obtaining dashboard from **host**/api/dashboards/uid/**uid**. Got Status 401 Unauthorized, message: {\"extra\":null,\"message\":\"Unauthorized\",\"messageId\":\"auth.unauthorized\",\"statusCode\":401,\"traceID\":\"\"}\n " pluginID=mahendrapaipuri-dashboardreporter-app
logger=context userId=3 orgId=1 uname=sa-grafana-reporter t=2024-04-16T07:11:21.22960438Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=10.0.0.2 time_ms=47 duration=47.020202ms size=24 referer= handler=/api/plugins/:pluginId/resources/*

Since Grafana recommends the use of service account for automated and scheduled routine, it may be a nice feature to add.

Answer 1 · 2024-04-16T09:33:44.000Z

Hello @and-mora

Cheers for creating the issue. Yes, I agree that we will need to support alternative auth methods.

As far as i understand, the only supported authentication method is the session cookie which usually happens through web browser.

Yes, this is correct, we just forward the cookie in the incoming request to make API calls to Grafana.

Since Grafana recommends the use of service account for automated and scheduled routine, it may be a nice feature to add.

This should be fairly straight forward to implement. Just to ensure we are on the same page, you make API requests with service account using Authorization header, isnt it? I assume same for the basic auth.

Answer 2 · 2024-04-16T10:11:29.000Z

sorry i didnt mention the command curl I used. For service account it must be used the Authorization bearer token:
curl --location "*host**/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report?dashUid=**uid**&from=now-1M%2FM&to=now-1M%2FM" -H "Authorization: Bearer **token**"
for basic auth it should be Authorization: Basic ....
To answer your question, yes, the forwarding of the Authorization header should be sufficient in both cases.

Answer 3 · 2024-04-18T14:07:46.000Z

hi @mahendrapaipuri, i continue here the discussion if you agree.
I still got the error on plugin API, i'll explain the steps i followed (with some context)

deploy grafana with plugin version of CI you pointed out
created a service account with viewer role
create a new token with no expire date for testing
run the curl (you can find it in the previous comment) and get 401 from the plugin when it calls the api/dashboards/uid
tested the report generation from browser and it works
tested the dashboard API with service account and it works

this is the log from grafana when failing curl:

logger=context userId=0 orgId=0 uname= t=2024-04-18T13:40:51.779462128Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/d0a30598-95a5-4098-89cc-435d965df315 status=401 remote_addr=10.0.0.2 time_ms=0 duration=330.282µs size=102 referer= handler=/api/dashboards/uid/:uid
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-04-18T13:40:51.780239292Z level=error msg="error generating report" endpoint=callResource err="error fetching dashboard d0a30598-95a5-4098-89cc-435d965df315: error obtaining dashboard from <host>:3001/api/dashboards/uid/d0a30598-95a5-4098-89cc-435d965df315. Got Status 401 Unauthorized, message: {\"extra\":null,\"message\":\"Unauthorized\",\"messageId\":\"auth.unauthorized\",\"statusCode\":401,\"traceID\":\"\"}\n " pluginID=mahendrapaipuri-dashboardreporter-app
logger=context userId=2 orgId=1 uname=sa-grafana-report t=2024-04-18T13:40:51.781152538Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=10.0.0.2 time_ms=172 duration=172.518565ms size=24 referer= handler=/api/plugins/:pluginId/resources/*

the problem might come from a miss-configuration of the plugin/server or may something gets wrong when grafana has https enabled (that's my case)? I didn't find many more differences with your docker-compose.
I can link the configuration I used if you want.
Thank you very much for your help and patience.

Answer 4 · 2024-04-18T16:52:03.000Z

That is strange error. How many Grafana orgs you have on your instance?

logger=context userId=0 orgId=0 uname= t=2024-04-18T13:40:51.779462128Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/d0a30598-95a5-4098-89cc-435d965df315 status=401 remote_addr=10.0.0.2 time_ms=0 duration=330.282µs size=102 referer= handler=/api/dashboards/uid/:uid

This first line in your log indicates that the plugin is sending request to orgId=0

logger=context userId=2 orgId=1 uname=sa-grafana-report t=2024-04-18T13:40:51.781152538Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=10.0.0.2 time_ms=172 duration=172.518565ms size=24 referer= handler=/api/plugins/:pluginId/resources/*

This last line indicates that the service account belongs to orgId=1. I am not quite sure how the API resources between different orgs in Grafana are managed though. I will investigate that.

In my local docker env, even with TLS, yes, I can generate reports both from service account and browser. However, I am always testing with a single organization though.

Answer 5 · 2024-04-19T07:17:28.000Z

@and-mora I have made a quick test with multiple orgs and it is working as expected. Just to be sure could you retry passing orgId in the query params to the report API and see if it works?

Also, could you please share more logs since the beginning of plugin loading if the problem still persists?

Cheers!

Answer 6 · 2024-05-01T15:28:41.000Z

Hello @and-mora, In fact it is a bug and I can confirm it wasnt working with service accounts. Could you please try this patch? You can download the plugin artifacts from here. The service account token must be configured for the plugin as well as stated in docs. If you are using grafana >= 10.3.0, you can use externalServiceAccounts feature which is stated in docs as well.

Answer 7 · 2024-05-01T19:59:00.000Z

hi @mahendrapaipuri, sorry i had no time to test again in the last days.
I'll keep up with the docs and test the plugin asap, thank you.

Answer 8 · 2024-05-01T22:22:42.000Z

I tested the plugin with grafana-oss:10.1.0, configured the service account token in the plugin settings and it works correctly.

Then i updated to grafana-oss:10.4.1 with the feature flag externalServiceAccounts enabled.

The plugin created a service account (extsvc-mahendrapaipuri-dashboardreporter-app) at startup with one token with no expiration date. As far as i understood when the feature flag is enabled there is no need to manually provision/configure the token in the plugin settings.
I created another service account to authenticate against the grafana server with API
run the curl and got error. Here something strange happens with the redirection url to grafana server. Let me explain.

logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-05-01T22:04:47.419240557Z level=error msg="error generating report" err="error fetching dashboard d0a30598-95a5-4098-89cc-435d965df315: error obtaining dashboard from <host>:3000/api/dashboards/uid/d0a30598-95a5-4098-89cc-435d965df315. Got Status 401 Unauthorized, message: {\"message\":\"Unauthorized\",\"traceID\":\"\"}\n " pluginID=mahendrapaipuri-dashboardreporter-app endpoint=callResource
logger=context userId=3 orgId=1 uname=sa-1-grafana-reporter t=2024-05-01T22:04:47.419894801Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=10.0.0.2 time_ms=276 duration=276.165659ms size=24 referer= handler=/api/plugins/:pluginId/resources/* status_source=downstream

here provisioning/plugins/app.yaml

apiVersion: 1

apps:
  - type: mahendrapaipuri-dashboardreporter-app
    org_id: 1
    org_name: Main Org.
    disabled: false
    jsonData:
      appUrl: <host>:3001
      [...]

the problem here is the port. I have configured the port 3001 but the plugin call the 3000 (on 3000 there is my stable grafana server and that's why i get 401 unauthorized, on 3001 the one I am using for these tests. Atm I don't know if it's due to new plugin or to grafana update.
Is the url to reach grafana retrieved from jsonData.appUrl?

Answer 9 · 2024-05-02T07:30:46.000Z

Cheers @and-mora for testing and detailed test report. Appreciate it!

As far as i understood when the feature flag is enabled there is no need to manually provision/configure the token in the plugin settings.

That is correct. When externalServiceAccounts feature flag is enabled, a service account and token will be automatically provisioned with scopes and permissions defined in plugin.

Is the url to reach grafana retrieved from jsonData.appUrl?

Yes, if the plugin is provisioned, we pick up the appUrl from provisioned config. The PR did not really change that part of code. Could you also please re-check if your test instance (the one on port 3001) is picking up the provisioned config from the correct directory?

The plugin will emit a log line with current provisioned config normally. Could you please look into that and see what it sees as appUrl?

Thanks a lot!!

Answer 10 · 2024-05-02T08:35:45.000Z

with Grafana 10.4.1:

test with externalServiceAccounts disabled; service account manually created and configured in the plugin settings: everything works fine.
test with externalServiceAccounts enabled: i got the wrong port configured

logger=context userId=1 orgId=1 uname=admin t=2024-05-02T08:24:50.313801028Z level=info msg="Request Completed" method=POST path=/api/serviceaccounts/ status=201 remote_addr=10.0.0.2 time_ms=31 duration=31.245116ms size=137 referer=<host>:3001/org/serviceaccounts/create handler=/api/serviceaccounts/ status_source=server
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-05-02T08:25:08.592480726Z level=info msg="Provisioned config" persistData=false dataPath= orientation=portrait dashboardMode =default layout=simple pluginID=mahendrapaipuri-dashboardreporter-app appUrl=<host>:3001 endpoint=callResource maxRenderWorkers=2 skipTlsCheck=false
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-05-02T08:25:08.594159297Z level=warn msg="failed to walk through grafana-image-renderer data dir" pluginID=mahendrapaipuri-dashboardreporter-app endpoint=callResource err="lstat /var/lib/grafana/plugins/grafana-image-renderer: no such file or directory"
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-05-02T08:25:08.793265704Z level=error msg="error generating report" endpoint=callResource err="error fetching dashboard d0a30598-95a5-4098-89cc-435d965df315: error obtaining dashboard from <host>:3000/api/dashboards/uid/d0a30598-95a5-4098-89cc-435d965df315. Got Status 401 Unauthorized, message: {\"message\":\"Unauthorized\",\"traceID\":\"\"}\n " pluginID=mahendrapaipuri-dashboardreporter-app
logger=context userId=3 orgId=1 uname=sa-1-grafana-reporter t=2024-05-02T08:25:08.793938468Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=10.0.0.2 time_ms=238 duration=238.043371ms size=24 referer= handler=/api/plugins/:pluginId/resources/* status_source=downstream

1st row: service account created
2nd row: provisioning correct config for plugin
3rd row: cannot find the renderer plugin (it is deployed as a separate service)
4th row: calling the wrong port

$ docker exec fd8 cat /etc/grafana/provisioning/plugins/app.yaml
apiVersion: 1

apps:
  - type: mahendrapaipuri-dashboardreporter-app
    org_id: 1
    org_name: Main Org.
    disabled: false
    jsonData:
      appUrl: <host>:3001
      [...]

between the 2 tests i only changed the grafana.ini file, commenting the enable line. Is there something else am i missing?
It looks like when the feature flag is enabled the configuration is not correctly read or set.

Answer 11 · 2024-05-02T09:20:29.000Z

When we enable externalServiceAccounts feature flag, Grafana will set GF_APP_URL env variable to the plugin process and seems like it is setting correct URL. The plugin will always override appUrl set in provisioned config when GF_APP_URL is found.

Could you please enable debug logging and rerun the service with externalServiceAccounts enabled so that we will see if GF_APP_URL is set to correct one? Also, if it is possible, could you share your docker setup environment? Another thing you can do is spawn a shell in container, look into env vars of plugin process by looking at cat /proc/<pid>/environ.

Answer 12 · 2024-05-02T10:02:16.000Z

with externalServiceAccounts enabled:

$ cat /proc/15/environ
GF_ENTERPRISE_APP_URL=<host>:3000/
GF_APP_URL=<host>:3000/

Dockerfile

ARG GRAFANA_VERSION=10.4.1@sha256:753bbb971071480d6630d3aa0d55345188c02f39456664f67c1ea443593638d0

FROM grafana/grafana-oss:${GRAFANA_VERSION}

ARG GF_INSTALL_IMAGE_RENDERER_PLUGIN="false"

ARG GF_GID="0"

ENV GF_PATHS_PLUGINS="/var/lib/grafana-plugins"
ENV GF_PLUGIN_RENDERING_CHROME_BIN="/usr/bin/chrome"

USER root

# add chromium and plugin for dashboard reporting
RUN mkdir -p "$GF_PATHS_PLUGINS" && \
    chown -R grafana:${GF_GID} "$GF_PATHS_PLUGINS" && \
    apk add --no-cache udev ttf-opensans chromium && \
    ln -s /usr/bin/chromium-browser "$GF_PLUGIN_RENDERING_CHROME_BIN"
#    grafana cli \
#        --pluginsDir "$GF_PATHS_PLUGINS" \
#        --pluginUrl ./plugin-artifacts.zip \
#        plugins install mahendrapaipuri-dashboardreporter-app;

ARG GF_INSTALL_PLUGINS=""

RUN if [ ! -z "${GF_INSTALL_PLUGINS}" ]; then \
      OLDIFS=$IFS; \
      IFS=','; \
      set -e ; \
      for plugin in ${GF_INSTALL_PLUGINS}; do \
        IFS=$OLDIFS; \
        if expr match "$plugin" '.*\;.*'; then \
          pluginUrl=$(echo "$plugin" | cut -d';' -f 1); \
          pluginInstallFolder=$(echo "$plugin" | cut -d';' -f 2); \
          grafana cli --pluginUrl ${pluginUrl} --pluginsDir "${GF_PATHS_PLUGINS}" plugins install "${pluginInstallFolder}"; \
        else \
          grafana cli --pluginsDir "${GF_PATHS_PLUGINS}" plugins install ${plugin}; \
        fi \
      done \
    fi

USER grafana

COPY ./provisioning /etc/grafana/provisioning
COPY ./dashboards /etc/grafana/dashboards
COPY ./grafana.ini /etc/grafana/grafana.ini

docker-compose.yml

version: "3.9"
services:
  grafana:
    image: grafana-test
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: on-failure
      update_config:
        failure_action: rollback
        order: start-first
    environment:
      GF_RENDERING_SERVER_URL: http://renderer:8081/render
      GF_RENDERING_CALLBACK_URL: <host>:3001
    ports:
      - '3001:3000'
    volumes:
# TLS stuff
      - key.pem:/etc/grafana/grafana.key
      - cert.pem:/etc/grafana/grafana.crt
      - ./dist:/var/lib/grafana-plugins/mahendrapaipuri-dashboardreporter-app
#      - storage:/var/lib/grafana no storage for tests
    networks: [...]

#  renderer:
#    image: grafana/grafana-image-renderer:3.10.1
#    environment:
#      # Recommendation of grafana-image-renderer for optimal performance
#      # https://grafana.com/docs/grafana/latest/setup-grafana/image-rendering/#configuration
#      - RENDERING_MODE=clustered
#      - RENDERING_CLUSTERING_MODE=browser
#      - RENDERING_CLUSTERING_MAX_CONCURRENCY=5
#      - RENDERING_CLUSTERING_TIMEOUT=60
#    ports:
#      - '8081'
#    networks: [...]

networks: [...]

next thing i'll enable the debug logging level and return to you soon

Answer 13 · 2024-05-02T10:23:41.000Z

grafana.ini in case it can be useful

[server]
http_addr =
http_port = 3000
domain = <host>
cert_key = /etc/grafana/grafana.key
cert_file = /etc/grafana/grafana.crt
enforce_domain = False
protocol = https

[smtp]
[...]

[plugins]
allow_loading_unsigned_plugins = mahendrapaipuri-dashboardreporter-app

[feature_toggles]
enable = externalServiceAccounts

here the debug logs:

logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-05-02T10:21:06.428049252Z level=info msg="Provisioned config" appUrl=<host>:3001 maxRenderWorkers=2 persistData=false dataPath= layout=simple orientation=portrait skipTlsCheck=false endpoint=callResource dashboardMode=default pluginID=mahendrapaipuri-dashboardreporter-app
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-05-02T10:21:06.430114544Z level=debug msg="Using Grafana app URL from environment variable" GF_APP_URL=<host>:3000 endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-05-02T10:21:06.430304346Z level=debug msg="Using Grafana data path from environment variable" GF_PATHS_DATA=/var/lib/grafana endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app

the problem seems in the override from GF_APP_URL. Is this field read from grafana.ini?

Answer 14 · 2024-05-02T11:29:18.000Z

Thanks a lot for your patient testing and sharing your test env @and-mora

I am assuming you are always using port 3000 in the container and mapping it to different port on the host for your test instance. And I guess your <host> is something that is publicly accessible. If you change your port in grafana.ini to 3001 as well and use appropriate port mapping in compose file, it should work.

problem seems in the override from GF_APP_URL. Is this field read from grafana.ini?

Yes, when we enable externalServiceAccounts, Grafana will spawn plugin by setting GF_APP_URL env var fetched from grafana.ini file.

Just out of curiosity, do you have a similar architecture in production? I mean configuring a publicly accessible domain URL directly on Grafana server and exposing different instances on different host ports?

Answer 15 · 2024-05-02T12:44:26.000Z

I am assuming you are always using port 3000 in the container and mapping it to different port on the host for your test instance. And I guess your is something that is publicly accessible

exactly right.

If you change your port in grafana.ini to 3001 as well and use appropriate port mapping in compose file, it should work.

I just tested it and it perfectly works! Much appreciation for your hints!

Just out of curiosity, do you have a similar architecture in production? I mean configuring a publicly accessible domain URL directly on Grafana server and exposing different instances on different host ports?

Yep, that's my cheap and rough production environment for a side project. At the moment there are different services exposed on different host ports, in the future i plan putting a reverse proxy with TLS termination (or a k8s ingress when i'll install k3s) in front of everything. :) if you have any suggestions you are welcome since I'm not a network expert

Answer 16 · 2024-05-02T13:00:00.000Z

I just tested it and it perfectly works! Much appreciation for your hints!

Awesome!!

Yep, that's my cheap and rough production environment for a side project. At the moment there are different services exposed on different host ports, in the future i plan putting a reverse proxy with TLS termination (or a k8s ingress when i'll install k3s) in front of everything. :) if you have any suggestions you are welcome since I'm not a network expert

It is fine the way you are doing it now as well but a reverse proxy with TLS termination would be a more conventional setup. In the case of reverse proxy you can run Grafana server on localhost and default port of 3000 for all the instances and that will make config simpler.