Datasource unusable without a clear error message (Plugin unavailable)
gillg opened this issue · 14 comments
What happened:
I can't explain in detail because I lack some logs, but I configured the datasource using an STS AssumeRole mecanism from a child account to the master organization AWS account.
I confirmed this mecanism because I tried to configure the same role with the "CloudWatch" datasource and it works like a charm (using the policy arn:aws:iam::aws:policy/service-role/AmazonGrafanaCloudWatchAccess).
So, initialy I just added the needed policies (recomended in this repo) to the role, then I tried to add the managed policy arn:aws:iam::aws:policy/service-role/AmazonGrafanaAthenaAccess, and even AdministratorAccess temporarly...
In 100% of the cases I get this error in UI and nothing in logs
I tried in different grafana version, initialy with different plugins version, but now I'm on the combo : Grafana 9.5.2 and Athena datasource 2.9.5 and still the same error.
What you expected to happen:
At least a clear error message... ?
How to reproduce it (as minimally and precisely as possible):
I would say install the plugin in the latest grafana version, but I guess it works for other people
The error apprears when I use the explorer, or when clicking on "Save & Test" in the datasource
Anything else we need to know?:
Exact same datasource config working, with the same role ARN for CloudWatch
Environment:
- Grafana version: 9.5.2
- Plugin version: 2.9.5
- OS Grafana is installed on: Amz Linux2 inside official grafana docker image
- User OS & Browser: Windows 10 / Chrome
@gillg hmm I agree that "plugin unavailable" is a very unhelpful message for us to share! So if we can figure out the root error, we should be able to make a ticket to make sure we catch/handle it properly! I'm trying to figure out how to recreate the error on my machine and so far unable to.
Some questions that might help:
- Are you able to see any backend logs that might reveal more information?
- Are there any logs in the browser console or errors in your network tab that might indicate what has gone wrong?
- In the screenshot you've posted above of the plugin config, I noticed that Datasource/Database/Workgroup are all blank. Are you able to select any thing there?
- Does uninstalling and reinstalling the athena plugin help at all? Does using an earlier version of plugin help?
Thanks @sarahzinger for your advices !
- I just tried to uninstall reinstall, same thing.
- The datasource / database & co can't be selected, it's when I click on it that I have the error.
- In the console after clicking on "Data source"
- Nothing special in network except the 503 error above
- I just tried to delete / recreate the datasource, same error.
- I don't have backend logs, but I guess they are synced to the stdout on docker.
logger=context userId=2 orgId=6 uname=********** t=2023-06-01T14:35:31.080290695Z level=warn msg="Plugin unavailable" error="plugin unavailable" traceID=
logger=context userId=2 orgId=6 uname=********** t=2023-06-01T14:35:31.080555345Z level=error msg="Request Completed" method=POST path=/api/datasources/27/resources/workgroups status=503 remote_addr=********** time_ms=30 duration=30.295874ms
size=46 referer=https://**********/connections/your-connections/datasources/edit/d542d886-4b70-1234-a244-182886fa4eb0 handler=/api/datasources/:id/resources/*
Interesting! Do you know are you able to recreate the same issues locally when you use Docker? the 503 makes me wonder if we have some kind of build issue here.
Do you mind sharing exactly how you built grafana and installed Athena (ex through the plugin catalog vs manually, etc)
Hum, I installed Grafana via the official docker image (docker.io/grafana/grafana) in a simple manual docker-compose.
The Athena datasource is installed from the plugin catalog in the admin interface.
I didn't tried localy yet.
Any potential new idea ?
I can't reproduce easily the sceanrio with an assume role localy so if you have any other idea / logs to enable / env vars to enable anything helping to diagnose I take it.
I compared the md5sum of the zip release x64 linux binary to the same from grafana plugin catalog and they are exactly the same.
Can we call the plugin binary in a certain way by commandline or whatever to make more in depth tests ?
Hi @gillg unfortunately I am unable to reproduce the error in my instance. When I install Athena from the plugin catalogue, it works fine with assume role or otherwise.
You mentioned you installed the plugin via the catalog. Can you try to remove it, then install it manually with these instructions?
Closing due to inactivity
I've found the reason randomly I think !
The executable can't run for a reason on my system.... But no idea why.
From host:
[root@xxxxxx ~]# ll /var/lib/grafana/plugins/grafana-athena-datasource/
total 30940
-rw-r--r-- 1 472 root 6558 Nov 20 14:19 CHANGELOG.md
drwxr-xr-x 2 472 root 59 Nov 20 14:19 dashboards
-rwxr-xr-x 1 472 root 1845 Nov 20 14:19 go_plugin_build_manifest
-rwxr-xr-x 1 472 root 30670848 Nov 20 14:19 gpx_athena_linux_amd64
drwxr-xr-x 2 472 root 22 Nov 20 14:19 img
-rw-r--r-- 1 472 root 11357 Nov 20 14:19 LICENSE
-rw-r--r-- 1 472 root 1745 Nov 20 14:19 MANIFEST.txt
-rw-r--r-- 1 472 root 210241 Nov 20 14:19 module.js
-rw-r--r-- 1 472 root 727287 Nov 20 14:19 module.js.map
-rw-r--r-- 1 472 root 1531 Nov 20 14:19 plugin.json
-rw-r--r-- 1 472 root 36259 Nov 20 14:19 README.md
[root@xxxxx ~]# /var/lib/grafana/plugins/grafana-athena-datasource/gpx_athena_linux_amd64
-bash: /var/lib/grafana/plugins/grafana-athena-datasource/gpx_athena_linux_amd64: Permission denied
[root@xxxxxx ~]# uname -a
Linux ol-usw2-monitor-gra 4.14.314-237.533.amzn2.x86_64 #1 SMP Thu May 4 09:55:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@xxxxxx ~]# file /var/lib/grafana/plugins/grafana-athena-datasource/gpx_athena_linux_amd64
/var/lib/grafana/plugins/grafana-athena-datasource/gpx_athena_linux_amd64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
[root@xxxxxx ~]# ldd /var/lib/grafana/plugins/grafana-athena-datasource/gpx_athena_linux_amd64
ldd: warning: you do not have execution permission for `/var/lib/grafana/plugins/grafana-athena-datasource/gpx_athena_linux_amd64'
not a dynamic executable
[root@xxxxxx ~]# sestatus
SELinux status: disabled
Inside the container:
5fa62915fc64:/usr/share/grafana# uname -a
Linux 5fa62915fc64 4.14.314-237.533.amzn2.x86_64 #1 SMP Thu May 4 09:55:30 UTC 2023 x86_64 Linux
5fa62915fc64:/usr/share/grafana# file /var/lib/grafana/plugins/grafana-athena-datasource/gpx_athena_linux_amd64
/var/lib/grafana/plugins/grafana-athena-datasource/gpx_athena_linux_amd64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=VcORba1xzH6Ij5B2Elir/KLXUtHGVw0wY-vwIORFe/6xNRtli0REAp-vq1vyAK/Y_QIZGB5gUUI1cip7Jqs, stripped
5fa62915fc64:/usr/share/grafana# ldd /var/lib/grafana/plugins/grafana-athena-datasource/gpx_athena_linux_amd64
/lib/ld-musl-x86_64.so.1: /var/lib/grafana/plugins/grafana-athena-datasource/gpx_athena_linux_amd64: Not a valid dynamic program
5fa62915fc64:/usr/share/grafana# cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.18.4
PRETTY_NAME="Alpine Linux v3.18"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
Do you think it could be related to missing libs in the Alpine image ?
Hi @gillg unfortunately I am unable to reproduce the error in my instance. When I install Athena from the plugin catalogue, it works fine with assume role or otherwise.
You mentioned you installed the plugin via the catalog. Can you try to remove it, then install it manually with these instructions?
Hi @gillg can you try installing the plugin from the instructions above (step 1 & 2) or with Grafana CLI?
I wonder if you are having the same issue with other Grafana external plugins? (Cloudwatch is bundled with Grafana, so I'm not surprised it works fine, unlike Athena)
Exact same result using GF_INSTALL_PLUGINS and uninstalling the plugin before relaunch.
I don't have any issue with other plugins but they don't have an embedded binary. They are pure Js usually.
@gillg one more though to try: when you uninstall/reinstall the athena plugin are you also deleting the datasources instances that reference the plugin? I wonder if you:
- uninstall athena
- delete any old datasource configurations referencing athena
- reinstall athena
- and then create a fresh configuration if you'll see the same error message?
Other than that I'm not really sure.
Here's my repro steps, let me know if I'm missing something:
- ran
docker run -d -p 3000:3000 --name=grafana grafana/grafana-oss - signed in and navigated to Connections
- searched for athena and clicked install
- then click add new datasource
- then added my configuration I used access/keys because it was easy to make some quickly for a test env, but when I put in dummy data for the sdk I get more predictable errors about not having permission rather than "plugin unavailable"
I would guess it's something off about your docker setup. I see some talk of using bind mounts here: https://grafana.com/docs/grafana/next/setup-grafana/installation/docker/#use-bind-mounts-1 I wonder if that's relevant for you? Maybe if you can share more of your docker file (with any sensitive info removed) we can try to reproduce better
@sarahzinger Thanks for your detailed test !!!
I think I understand something now....
The image I use is https://hub.docker.com/r/grafana/grafana and not https://hub.docker.com/r/grafana/grafana-oss
The images sha256 seems the same but just in case... I will test.
My config:
version: '3.8'
services:
grafana:
container_name: grafana
image: grafana/grafana:9.5.14
ports:
- target: 3000
published: 3000
mode: host
volumes:
- /var/lib/grafana/:/var/lib/grafana/
restart: unless-stopped
environment:
- GF_SERVER_PROTOCOL=http
- GF_SERVER_HTTP_PORT=3000
# Could be false behind nginx
- GF_SERVER_ENABLE_GZIP=true
- GF_DATABASE_TYPE=mysql
- GF_FEATURE_TOGGLES_ENABLE=ngalert
- GF_UNIFIED_ALERTING_ENABLED=true # NGAlert alias alertmaanger
- GF_ALERTING_ENABLED=false # Legacy alerting
- GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=camptocamp-prometheus-alertmanager-datasource,agenty-flowcharting-panel,flant-statusmap-panel
- GF_INSTALL_PLUGINS=grafana-athena-datasource
- GF_PLUGINS_ENABLE_ALPHA=true
- GF_RENDERING_SERVER_URL=http://grafana-image-renderer:8081/render
- GF_RENDERING_CALLBACK_URL=http://grafana:3000/
#### Other options but not revelant here...
Sadly... not better but not surprised as the sha256 of the image is strictly the same :/