[BUG] Container is not starting anymore (Startup broken)

Question

[BUG] Container is not starting anymore (Startup broken)

tstrohmeier opened this issue 2 years ago · 19 comments

Since the PR #452 #452 was merged, new Kimai deployment can not start up because the dbtest.php check fails.

I guess this happens because this check here: https://github.com/tobybatch/kimai2/blob/main/assets/service.sh#L5-L11
was removed before this check is execute: https://github.com/tobybatch/kimai2/blob/main/assets/startup.sh#L14

Answer 1 · 2023-03-21T22:15:04.000Z

This issue is blocking our deployment of a Kimai instance.

To speed the fix up I created a PR for this bug: ~~#489~~ #490

Answer 2 · 2023-03-21T22:56:04.000Z

So you say there is an issue existing since two month and it shows up NOW and in exactly one installation?

This issue is blocking our deployment of a Kimai so therefore this issue is very URGENT.

It can't be urgent, because "it is broken since two month" and you post just now.
If you need "urgent help", contact us for professional support and we talk about pricing.

Answer 3 · 2023-03-21T23:51:56.000Z

@kevinpapst The PR was merged yesterday.

I created a own image with the fix now and it's working fine now, but still I think PR #452 introduced an bug yesterday.

Answer 4 · 2023-03-22T08:37:09.000Z

Ups, you are right. Posting too late in the night is never a good idea 😁

Anyway, the sentence:

This issue is blocking our deployment of a Kimai so therefore this issue is very URGENT.

makes a completely different impression than something like:

IMO this issue is blocking EVERYONE from an update / deployment of the Kimai container.

Which would trigger completely different (re-)actions.

@tobybatch can you confirm that your container is broken since the merge yesterday?

Answer 5 · 2023-03-22T08:53:15.000Z

To test this issue:
Change the image in this line https://github.com/tobybatch/kimai2/blob/main/compose/docker-compose.fpm.prod.yml#L37 to kimai/kimai2:fpm-1.30.11-prod

kimai/kimai2:fpm-1.30.11-prod is build with the new Dockerfile & Scripts: https://hub.docker.com/layers/kimai/kimai2/fpm-1.30.11-prod/images/sha256-e2b1eb59a37cf5220ed5f53429e1e54bd7a82451957329e15416ca8094fa4459?context=explore

kimai/kimai2:latest still has the old Dockerfile & scripts: https://hub.docker.com/layers/kimai/kimai2/latest/images/sha256-840ccc74942d9d2590ebfc168212e6477dfd24bb3e4beb8de85a7a8855602f7d?context=explore

Answer 6 · 2023-03-22T08:54:49.000Z

@kevinpapst

Anyway, the sentence:

This issue is blocking our deployment of a Kimai so therefore this issue is very URGENT.

makes a completely different impression than something like:

IMO this issue is blocking EVERYONE from an update / deployment of the Kimai container.

Which would trigger completely different (re-)actions.

Yes it was a bit late and I got panic. I am sorry for that.

Answer 7 · 2023-03-22T09:04:38.000Z

A other issue from this PR: In the new startup.sh the handleStartup function is never called.

Answer 8 · 2023-03-22T09:10:41.000Z

And one more issue: After fixing the db-test and adding the handleStartup to the startup.sh locally, I get a HTTP Status 500. This is related to the changed handling of the user in the container.

The user & group www-data has the ID 82

cc8ecd9df3ca:/opt/kimai# id -u www-data
82
cc8ecd9df3ca:/opt/kimai# id -g www-data
82

As you can see here the new introduced user www-kimai has the ID 1000 is not owning the files in /opt/kimai

cc8ecd9df3ca:/opt/kimai# ls -al
total 1160
drwxr-xr-x  1 www-data  www-data   4096 Mar 22 10:04 .
drwxr-xr-x  1 root      root       4096 Mar 22 10:03 ..
-rw-r--r--  1 www-data  www-data    534 Mar 22 10:01 .codecov.yml
-rw-r--r--  1 www-data  www-data    240 Mar 22 10:01 .editorconfig
-rw-r--r--  1 www-data  www-data    821 Mar 22 10:04 .env
-rw-r--r--  1 www-data  www-data    821 Mar 22 10:01 .env.dist
drwxr-xr-x  1 www-data  www-data   4096 Mar 22 10:01 .git
drwxr-xr-x  1 www-data  www-data   4096 Mar 22 10:01 .github
-rw-r--r--  1 www-data  www-data    237 Mar 22 10:01 .github_changelog_generator
-rw-r--r--  1 www-data  www-data   1023 Mar 22 10:01 .gitignore
-rw-r--r--  1 www-data  www-data   5969 Mar 22 10:01 .php-cs-fixer.dist.php
-rw-r--r--  1 www-data  www-data     82 Mar 22 10:01 CHANGELOG.md
-rw-r--r--  1 www-data  www-data   1235 Mar 22 10:01 CONTRIBUTING.md
-rw-r--r--  1 www-data  www-data   1100 Mar 22 10:01 LICENSE
-rw-r--r--  1 www-data  www-data   5560 Mar 22 10:01 README.md
-rw-r--r--  1 www-data  www-data   1070 Mar 22 10:01 SECURITY.md
-rw-r--r--  1 www-data  www-data  22136 Mar 22 10:01 UPGRADING.md
drwxr-xr-x  1 www-data  www-data   4096 Mar 22 10:01 assets
drwxr-xr-x  1 www-data  www-data   4096 Mar 22 10:01 bin
-rw-r--r--  1 www-data  www-data   6670 Mar 22 10:04 composer.json
-rw-r--r--  1 www-data  www-data 528729 Mar 22 10:04 composer.lock
drwxr-xr-x  1 www-data  www-data   4096 Mar 22 10:01 config
-rw-r--r--  1 www-data  www-data   1353 Mar 22 10:04 package.json
-rw-r--r--  1 www-data  www-data  73455 Mar 22 10:04 php-cli.ini
-rw-r--r--  1 www-data  www-data   1019 Mar 22 10:01 phpstan.neon
-rw-r--r--  1 www-data  www-data   3192 Mar 22 10:01 phpunit.xml.dist
drwxr-xr-x  6 www-data  www-data   4096 Mar 22 10:04 public
drwxr-xr-x  1 www-data  www-data   4096 Mar 22 10:01 src
-rw-r--r--  1 www-data  www-data  19580 Mar 22 10:01 symfony.lock
drwxr-xr-x  1 www-data  www-data   4096 Mar 22 10:01 templates
drwxr-xr-x  1 www-data  www-data  32768 Mar 22 10:01 translations
drwxr-xr-x  9 www-kimai     1000   4096 Mar 22 10:06 var
drwxr-xr-x 51 www-data  www-data   4096 Mar 22 10:04 vendor
-rw-r--r--  1 root      root          8 Mar 22 10:04 version.txt
-rw-r--r--  1 www-data  www-data   2173 Mar 22 10:01 webpack.config.js
-rw-r--r--  1 www-data  www-data 325693 Mar 22 10:01 yarn.lock

This leads to this error:

kimai_1 | 172.20.0.4 - 22/Mar/2023:10:07:21 +0100 "GET /index.php" 500

Was #452 ever tested with fpm-prod images?

Answer 9 · 2023-03-22T10:47:05.000Z

I've merged your fixing PR, thanks.

I've been working on a new CI process to build multiarch images and I think it got accidentally merged into the main branch before I meant to. Sorry

Your PR should fix the start up script. The chown on the files got backed out 2 months ago. We've had a lot of requests for setting UID/GID at run time. The broke some existing installs so got removed.

Answer 10 · 2023-03-22T10:47:45.000Z

And I've re-triggered the GH action to rebuild the images

Answer 11 · 2023-03-22T11:06:37.000Z

@tobybatch The next PR: #491

Like mentioned above the handleStartupmethod is not called in startup.sh. This has two effects:

No public folder because tar -zx -C /opt/kimai -f /var/tmp/public.tgz is not called
The new User logic is not executed

Can you please review this PR?
Otherwise the new images will still not work.

Answer 12 · 2023-03-22T13:47:04.000Z

Otherwise the new images will still not work.

I can confirm, the current image kimai/kimai2:apache only leaves me with an error 500 when I visit the page...
I can also see this in the docker startup process:

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /opt/kimai/var/cache/pro_/twig/cf/cfb6058a60f6c218d91aba594fd3e2d6.php on line 637
PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /opt/kimai/vendor/symfony/var-dumper/Cloner/VarCloner.php on line 133

I had the same issue when running /opt/kimai/bin/console kimai:update in the container directly. php -d memory_limit=512M /opt/kimai/bin/console kimai:update did work.

Answer 13 · 2023-03-22T13:53:20.000Z

Running this from #491 in my container revived my system (until it will hopefully be fixed by a new image, @tobybatch 😉):

chown -R $USER_ID:$GROUP_ID /opt/kimai
chown -R $USER_ID:$GROUP_ID /usr/local/etc/php/php.ini

Thank you @tstrohmeier for the fix and @tobybatch for all the work!

Answer 14 · 2023-03-22T15:09:02.000Z

Some thing which is still odd: The chown changes the ownership, but after the php-fpm process started the ownership of the folder/opt/kimai/var/cache/prod is changed back to root again.
Does anyone has an idea why?

Answer 15 · 2023-03-23T10:33:26.000Z

@tobybatch

Your PR should fix the start up script. The chown on the files got backed out 2 months ago. We've had a lot of requests for setting UID/GID at run time. The broke some existing installs so got removed.

Setting the UID/GID at runtime still causes some issues:

[2023-03-23 10:55:32] request.CRITICAL: Uncaught PHP Exception Symfony\Component\Filesystem\Exception\IOException: "Failed to create "/opt/kimai/var/cache/prod/translations": mkdir(): Permission denied" at /opt/kimai/vendor/symfony/filesystem/Filesystem.php line 105 {"exception":"[object] (Symfony\\Component\\Filesystem\\Exception\\IOException(code: 0): Failed to create \"/opt/kimai/var/cache/prod/translations\": mkdir(): Permission denied at /opt/kimai/vendor/symfony/filesystem/Filesystem.php:105)"} []

[2023-03-23 10:55:46] request.CRITICAL: Uncaught PHP Exception Twig\Error\RuntimeError: "An exception has been thrown during the rendering of a template ("Failed to create "/opt/kimai/var/cache/prod/translations": mkdir(): Permission denied")." at /opt/kimai/templates/timesheet/layout-listing.html.twig line 40 {"exception":"[object] (Twig\\Error\\RuntimeError(code: 0): An exception has been thrown during the rendering of a template (\"Failed to create \"/opt/kimai/var/cache/prod/translations\": mkdir(): Permission denied\"). at /opt/kimai/templates/timesheet/layout-listing.html.twig:40, Symfony\\Component\\Filesystem\\Exception\\IOException(code: 0): Failed to create \"/opt/kimai/var/cache/prod/translations\": mkdir(): Permission denied at /opt/kimai/vendor/symfony/filesystem/Filesystem.php:105)"} []

As mentioned above:

Some thing which is still odd: The chown changes the ownership, but after the php-fpm process started the ownership of the folder/opt/kimai/var/cache/prod is changed back to root again.
Does anyone has an idea why?

The ownership is changed after the php-fpm process starts. This maybe happens because after the PR #452 the User in the Dockerfile was removed and now the container is executed as root:

kimai-6b74d4b44c-85p4g:/opt/kimai/var/cache# ls -al
total 16
drwxr-xr-x 4 www-data www-data 4096 Mar 23 09:55 .
drwxr-xr-x 9 www-data www-data 4096 Mar 23 09:55 ..
-rw-r--r-- 1 www-data www-data    0 Mar 23 09:42 .gitkeep
drwxr-xr-x 2 root     root     4096 Mar 23 09:55 pro_
drwxr-xr-x 7 root     root     4096 Mar 23 09:55 prod

@tobybatch Is it really a good idea to set the user at runtime?

Answer 16 · 2023-03-23T10:35:54.000Z

It is my most common feature request. It comes from people who want to mount local folders into the container rather than volumes. I'm going to fix this and the memory limit part today. I'll get a fix out as soon as possible.

Answer 17 · 2023-03-23T10:37:09.000Z

Ok I see. Thank you!

Answer 18 · 2023-03-23T11:06:58.000Z

It is my most common feature request. It comes from people who want to mount local folders into the container rather than volumes. I'm going to fix this and the memory limit part today. I'll get a fix out as soon as possible.

Thank you very much!

Answer 19 · 2023-03-27T08:40:33.000Z

New images are now built and pushed (including multiarch). I'll close this for now, please open a new ticket if issues persist.