Provide ab fat zip with plugin and the chromium binary
jkroepke opened this issue · 16 comments
Would it be possible to setup an fat zip with plugin itself and the chromium binary?
Originally posted by @jkroepke in #41 (comment)
My proposal would be a dedicated repository which takes the responsibility to building an chrome binary which works out-of-the-box on grafana containers.
If you agree, can could take care of that, providing linux (amd64 and arm64 builds) as release.
On new plugin releases, the dashboard-reporter could download the release artifact and embedded into the zip file.
What did you think about?
Hello @jkroepke Thanks a lot for your interest.
Before, we need to do some tests locally to see if it works with just providing pre-compiled binary of chromium. I will try to do some tests. If it works, that would be awesome and we can discuss more on how to bundle the chromium into plugin releases. What do you think?
The plugin already looks into the grafana-image-renderer plugin, if there is an local chrome binary.
In general, using a chrome plugin from a plugin may works.
Let me know, if you need assistance with building chromium.
Yes, we do use the chromium binary shipped with grafana-image-renderer when found. But if you look into docs, they explicitly specify that the shipped binary depends on shared libraries that must be installed on the system. This is why I am not sure if it works out-of-the-box.
@jkroepke I did a very rapid test. Used the latest Grafana image, installed grafana-image-renderer plugin in the container and attempted to use Print to PDF API call. This is the result:
db7e71d412a5:/var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64$ ./chrome --headless --disable-gpu --print-to-pdf https://www.chromestatus.com/
./chrome: error while loading shared libraries: libgobject-2.0.so.0: cannot open shared object file: No such file or directory
db7e71d412a5:/var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64$ ls -la
total 274076
drwxr-xr-x 7 grafana root 4096 Jun 10 07:26 .
drwxr-xr-x 3 grafana root 4096 Jun 10 07:26 ..
-rw-r--r-- 1 grafana root 248 Jun 10 07:26 ABOUT
drwxr-xr-x 2 grafana root 4096 Jun 10 07:26 MEIPreload
drwxr-xr-x 3 grafana root 4096 Jun 10 07:26 WidevineCdm
-rwxr-xr-x 1 grafana root 245397448 Jun 10 07:26 chrome
-rwxr-xr-x 1 grafana root 5061 Jun 10 07:26 chrome-wrapper
-rw-r--r-- 1 grafana root 679904 Jun 10 07:26 chrome_100_percent.pak
-rw-r--r-- 1 grafana root 1061647 Jun 10 07:26 chrome_200_percent.pak
-rwxr-xr-x 1 grafana root 1297832 Jun 10 07:26 chrome_crashpad_handler
-rwxr-xr-x 1 grafana root 212472 Jun 10 07:26 chrome_sandbox
drwxr-xr-x 2 grafana root 4096 Jun 10 07:26 hyphen-data
-rw-r--r-- 1 grafana root 10468208 Jun 10 07:26 icudtl.dat
-rwxr-xr-x 1 grafana root 245112 Jun 10 07:26 libEGL.so
-rwxr-xr-x 1 grafana root 6940632 Jun 10 07:26 libGLESv2.so
-rwxr-xr-x 1 grafana root 4134616 Jun 10 07:26 libvk_swiftshader.so
-rwxr-xr-x 1 grafana root 615272 Jun 10 07:26 libvulkan.so.1
drwxr-xr-x 2 grafana root 4096 Jun 10 07:26 locales
-rw-r--r-- 1 grafana root 2772 Jun 10 07:26 product_logo_48.png
drwxr-xr-x 4 grafana root 4096 Jun 10 07:26 resources
-rw-r--r-- 1 grafana root 8785537 Jun 10 07:26 resources.pak
-rw-r--r-- 1 grafana root 667625 Jun 10 07:26 v8_context_snapshot.bin
-rw-r--r-- 1 grafana root 107 Jun 10 07:26 vk_swiftshader_icd.json
-rwxr-xr-x 1 grafana root 37394 Jun 10 07:26 xdg-mime
-rwxr-xr-x 1 grafana root 33273 Jun 10 07:26 xdg-settings
This is what I was suspecting. Do you think we can ship a "standalone" instance of chromium with all shared libraries needed?
Did you do a ldd on the chrome binary? Running grafana inside docker container and install grafana-image-renderer brings a non-functional chromium installation.
/usr/share/grafana $ ldd /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome
/lib64/ld-linux-x86-64.so.2 (0x700958141000)
libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x700958141000)
libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x700958141000)
Error loading shared library libgobject-2.0.so.0: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libglib-2.0.so.0: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libnss3.so: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libnssutil3.so: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libsmime3.so: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libnspr4.so: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libdbus-1.so.3: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libatk-1.0.so.0: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libatk-bridge-2.0.so.0: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libcups.so.2: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libgio-2.0.so.0: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libdrm.so.2: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libatspi.so.0: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libexpat.so.1: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x700958141000)
Error loading shared library libX11.so.6: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libXcomposite.so.1: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libXdamage.so.1: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libXext.so.6: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libXfixes.so.3: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libXrandr.so.2: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libgbm.so.1: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libxcb.so.1: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libxkbcommon.so.0: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libpango-1.0.so.0: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libcairo.so.2: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
Error loading shared library libasound.so.2: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7009494d6000)
libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x700958141000)
Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-127.0.6496.0/chrome-linux64/chrome)
I already test that before open the issue. And grafana-image-renderer does not support linux/arm64.
Do you think we can ship a "standalone" instance of chromium with all shared libraries needed?
I guess yes.
Yes, I did check ldd on chromium binary and noticed missing libs.
My initial idea was to use a similar strategy as grafana-image-renderer to ship the chromium binary and its dependencies. Now that we know it would not work, we need to come up another approach.
Do you have any leads on any projects that are building standalone chromium instances? I am afraid that this is no going to be very easy task to deploy and maintain.
Do you have any leads on any projects that are building standalone chromium instances? I am afraid that this is no going to be very easy task to deploy and maintain.
not yet, and before invest time in that topic, I would hear your opinion first.
I know that browserless and puppeteer proving browser binaries.
Well, I think it has an added value especially in k8s context as this avoids building custom images for end users and operators. I do not work with k8s a lot. In your experience, how much of hassle is it to add chromium in Grafana container? I see that there is postStart hook that can be used to mutate image without needing to build custom image. Do you think postStart hook is an ideal way to add chromium to the Grafana container?
I mean, if we need to ship chromium with plugin, we will need to ship all the dependencies. This what all major distros package managers apt, yum, apk, etc already doing when we install chromium. What is the added value of replicating the same package within the plugin? Do you see what I mean?
In your experience, how much of hassle is it to add
chromiumin Grafana container?
If you are just using upstream containers, it's an hassle to setup a dedicated docker build pipelines.
Do you think postStart hook is an ideal way to add chromium to the Grafana container?
Maybe, the issue here is that containers are not running as root. At runtime, the container does not have access to unix package managers. apk add chrome is only availible on build time context.
But I see that it would not necessary to add chrome into the plugin zip. it's sufficient to package chrome in a separator container and load the content during initContainer phrase.
The only question would be to create package that containers all necessary library. Building a chromium as static binary may helps here.
If you are just using upstream containers, it's an hassle to setup a dedicated docker build pipelines.
Totally agree with this one. This is very inconvenient.
Maybe, the issue here is that containers are not running as root. At runtime, the container does not have access to unix package managers. apk add chrome is only availible on build time context.
Oh yeah! Didnt think about the runtime user of the containers. Well, in that postStart hook might not work in all the cases.
The only question would be to create package that containers all necessary library. Building a chromium as static binary may helps here
Static binary with all dependencies means HUGE binary file. We need to think about RAM it eats for each invocation. This can be a big problem for k8s and most probably containers end up with OOM errors.
Static binary with all dependencies means HUGE binary file. We need to think about RAM it eats for each invocation. This can be a big problem for k8s and most probably containers end up with OOM errors.
Yeah, but I think this problem already exists anyways and could be only solved by #41
Yeah, but I think this problem already exists anyways and could be only solved by #41
No, I dont think so. The plugin itself calls chromium at most twice for a given API call. Even these two invocations of chromium are very "light". Most of the heavy lifting is done by grafana-image-renderer to generate PNGs of each panel in the dashboard.
The question here is memory footprint of chromium that loads shared libs dynamically vs memory footprint of static built of chromium with all dependencies for a simple invocation in headless mode. chromium binary shipped by playwright on my ubuntu workstation is already > 350 MB. And this binary uses the shared libs from system. If we need to include all those libs in the binary, it will bloat to at least 600 - 700 MB.
I guess we need to aim at shipping chromium with all the shared libs and a wrapper script to chrome that manipulates the LD_LIBRARY_PATH at runtime to look for shared libs at custom dir. Does it make sense to you?
Why did you think that the memory footprint is different?
Mention that grafana-image-renderer and grafana running on different containers and not sharing memory
Why did you think that the memory footprint is different?
Imagine there are 10 concurrent report generation API calls to the reporter plugin. With dynamically linked chromium, all processes will share the same loaded dynamic libs. Whereas in the case of statically linked chromium, all those dynamic libs (although only functions that used in libs) will be replicated across each process which will increase memory footprint. Does it make sense?
Mention that grafana-image-renderer and grafana running on different containers and not sharing memory
Exactly. That is why we are not worried about memory foot print of grafana-image-renderer. If it crashes due to OOM, the renderer will be unavailable for few seconds until k8s will recreate a new container. However, we are stuck with reporter plugin running in the same container as Grafana for the moment. So, if reporter plugin uses too much memory, there is risk that it will bring down Grafana instance with it which we dont want.
TLDR; we have to minimise the memory footprint of reporter plugin as it risks crashing whole Grafana instance. We dont need to worry about memory footprint of grafana-image-renderer as it is capable of running in separate container, so it does not effect Grafana container.
Do you agree?
I will try to archive something with the binaries provided by the chromedp project. I remember, its the library that this project is using to control chrome anyways.
They provide optimized chrome version, having few libraries.
root@911a800a25f1:/# ldd headless-shell/headless-shell
linux-vdso.so.1 (0x0000ffffba073000)
libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffffafb80000)
libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffffafb50000)
libnss3.so => /lib/aarch64-linux-gnu/libnss3.so (0x0000ffffaf9f0000)
libnssutil3.so => /lib/aarch64-linux-gnu/libnssutil3.so (0x0000ffffaf9a0000)
libnspr4.so => /lib/aarch64-linux-gnu/libnspr4.so (0x0000ffffaf940000)
libexpat.so.1 => /lib/aarch64-linux-gnu/libexpat.so.1 (0x0000ffffaf8f0000)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffaf850000)
libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffffaf810000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffaf660000)
/lib/ld-linux-aarch64.so.1 (0x0000ffffba036000)
libplc4.so => /lib/aarch64-linux-gnu/libplc4.so (0x0000ffffaf630000)
libplds4.so => /lib/aarch64-linux-gnu/libplds4.so (0x0000ffffaf600000)
and reduced total size (185MB uncompressed binary)